# Authentication & API keys (https://expand.ai/docs/authentication) Sign in for the CLI and local tools, use an API key everywhere else, and send it the right way for each surface. There are two ways to prove who you are to Expand. | Use | When | | ----------------------------- | -------------------------------------------------------------------------------------------------- | | Sign in with `expandai login` | Your own machine. The CLI and a local MCP server reuse the session, and there is no key to manage. | | An API key | CI, servers, scripts, the TypeScript SDK, direct API calls, and MCP clients that cannot sign in. | **Recommendation:** Keep API keys on servers and in CI. On your own machine, sign in instead, so no key sits in a config file. ## Sign in Sign in once and the CLI keeps the session on your machine: ```bash expandai login expandai whoami ``` `expandai login` offers two choices: open the browser and approve, or paste an API key. `expandai whoami` shows your account, organization, and how you are authenticated. `expandai logout` clears the stored credentials. A local MCP server started with `expandai mcp run` uses the same stored credentials as the CLI, so no API key goes into your agent's config. The hosted MCP server signs you in through the browser instead. See [Connect your agent (MCP)](https://expand.ai/docs/mcp-server). ## Create an API key Create a key in the [dashboard](https://expand.ai/settings/api-keys). Keys start with `xpnd_`. **Warning:** The raw key is shown once, when you create it. Store it straight away. ## Use the key The CLI, the TypeScript SDK, and direct API calls all read `EXPAND_API_KEY`: ```bash export EXPAND_API_KEY="xpnd_..." ``` How the key travels depends on where you call Expand: | Surface | How the key is sent | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | HTTP API | The `x-expand-api-key` header. | | TypeScript SDK | `new ExpandClient()` reads `EXPAND_API_KEY` and sends it as `x-expand-api-key`. Pass `apiKey` only when the key comes from somewhere else. | | CLI | `EXPAND_API_KEY` is used when there is no stored sign-in. | | Hosted MCP server | `Authorization: Bearer xpnd_...` in the client config. | **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://news.ycombinator.com' }) ``` **CLI** ```bash # The key is used when there is no stored sign-in. expandai fetch https://news.ycombinator.com ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://news.ycombinator.com"}' ``` **Note:** In the CLI, a stored sign-in from `expandai login` takes precedence over `EXPAND_API_KEY`. To use the key on a machine where you have signed in, run `expandai logout` first. For an MCP client with no browser sign-in, send the key as a bearer token: ```json { "mcpServers": { "expand": { "url": "https://api.expand.ai/v1/mcp", "headers": { "Authorization": "Bearer xpnd_your_api_key" } } } } ``` **Warning:** A config file that holds an API key is a secret. Treat it like one. ## Environment and config `EXPAND_API_KEY` is the only setting most setups need. The CLI's other environment variables and its config file location are on [CLI: Commands, Environment and config](https://expand.ai/docs/cli-commands#environment-and-config). ## Next steps - [Quickstart](https://expand.ai/docs/quickstart): run your first Fetch. - [Connect your agent (MCP)](https://expand.ai/docs/mcp-server): set up Expand in a coding agent. - [TypeScript SDK](https://expand.ai/docs/typescript-sdk): client options, including timeouts and retries. - [Limits](https://expand.ai/docs/limits): what one key can do at a time. --- # CLI commands (https://expand.ai/docs/cli-commands) Every expandai command, argument, flag, and default. Fetch a page, highlight it, read its Appendix, sign in, set up a coding agent, and clean up. `expandai` is the Expand command line. You give it a URL and get Markdown text back on stdout, or a JSON object with `--format json`. The same binary signs you in, runs the local MCP server, and installs the Expand Skill and hooks into coding agents. ```bash npm install -g expandai ``` Every command also runs without a global install, through `npx -y expandai `. ## Most common ```bash expandai fetch https://news.ycombinator.com expandai fetch https://www.figma.com/pricing/ --format json expandai fetch https://docs.stripe.com/webhooks --highlights "signature verification" expandai login expandai whoami ``` ## Command index | Command | Definition | | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `expandai fetch ` | Fetch a page and print Markdown text. `--format json` prints a JSON object. | | `expandai fetch --highlights ` | Fetch a page and print Fetch Highlights for the query. | | `expandai snapshots get ` | Read the public Snapshot catalog. | | `expandai snapshots get --highlights ` | [Snapshot Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight#snapshot-highlights) on a page you already fetched. No new Fetch. | | `expandai appendix ` | Read the [Appendix](https://expand.ai/docs/fetch#appendix) of a page you already fetched. No new Fetch. | | `expandai login` | Sign in with the browser, or store an API key. | | `expandai whoami` | Show the account, organization, and credential the CLI is using. | | `expandai logout` | Clear stored credentials. | | `expandai init` | Sign in and configure the coding agents on this machine. | | `expandai status` | Report what is configured for one agent and scope. Changes nothing. | | `expandai mcp run` | Run the Expand MCP server over stdio. | | `expandai mcp install ` | Register the MCP server in one agent's config. | | `expandai skill install ` | Copy the Expand Skill into one agent's skills directory. | | `expandai hook install ` | Install the Skill, the MCP server, and the web-fetch hook for one agent. | | `expandai uninstall [target] [host] [scope]` | Remove hooks, Skills, and MCP entries. | | `expandai telemetry status` | Show whether anonymous usage telemetry is on. | | `expandai crawl ` | Mirror a site into a folder of Markdown. | **Note:** Batched Fetch has no CLI command. Start and poll a run through the [TypeScript SDK](https://expand.ai/docs/typescript-sdk#batched-fetch) or the API. See [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch). ## Fetch ### `expandai fetch ` Fetch a page and print it. ```bash expandai fetch https://www.figma.com/pricing/ ``` | Argument | Definition | | -------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `url` | The page to fetch. `http` or `https` only. Input without a scheme gets `https://` added. The hostname needs a dot, or is `localhost`. | | Option | Definition | Default | | --------------- | ------------------------------------------------------------------------------ | ------- | | `--format` | `text` prints Markdown text. `json` prints a JSON object. | `text` | | `--highlights` | A query. Prints Fetch Highlights for it instead of the page. | None | | `--max-results` | Most snippets to print when `--highlights` is set, `1` to `50`. | `5` | | `--min-score` | Lowest relevance score a snippet needs when `--highlights` is set, `0` to `1`. | `0.6` | **Note:** The CLI has no `--include` flag, so the [include options](https://expand.ai/docs/fetch/fetching-a-url#include-options) stay at their defaults. To choose layers, use the TypeScript SDK or the API. Without `--highlights`, `--format text` is [Markdown Mode](https://expand.ai/docs/fetch/fetching-a-url#two-modes): the CLI calls `POST /v1/fetch` and prints the Markdown text, frontmatter first. `--format json` is JSON Mode: the CLI calls `POST /v1/fetch/json` and prints the JSON object, with the [Main Markdown](https://expand.ai/docs/fetch#main-markdown) in `markdown`, the [State JSON](https://expand.ai/docs/fetch#state-json) in `json`, and the Snapshot in `meta`. ```bash expandai fetch https://www.figma.com/pricing/ --format json ``` **Note:** When the Fetch returns no Markdown, the command fails with `No markdown content was returned`. When the origin never answered, `--format text` fails with the reason, and `--format json` prints the origin error object instead of content. See [Errors & retries](https://expand.ai/docs/errors-and-retries). ## Highlights Two commands return snippets. `expandai fetch --highlights` makes a new Fetch and highlights it in the same request. `expandai snapshots get` highlights a Snapshot you already have. How they differ, and how to tune `--max-results` and `--min-score`, is on [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight). ### `expandai fetch --highlights ` Fetch a page and print [Fetch Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight) for the query. ```bash expandai fetch https://docs.stripe.com/webhooks --highlights "signature verification" --max-results 5 ``` With `--highlights`, the CLI always calls `POST /v1/fetch/json`, whatever `--format` says. `--format text` prints each snippet with its source, its score, and a citation link. `--format json` prints the whole JSON object, with the snippets under `data.highlights.snippets` and a `citationUrl` added to each one. **Note:** Zero snippets is an error on the command line. `--format text` fails with `No snippets were returned`. Lower `--min-score` and try again. ### `expandai snapshots get ` Read the public Snapshot catalog without authentication. The default text output is Markdown; `--format json` returns the catalog as JSON. ```bash expandai snapshots get ``` Add `--highlights "query"` to get relevant passages. This requires authentication and costs one Highlights credit. ### `expandai snapshots get --highlights ` Print [Snapshot Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight#snapshot-highlights) for a page you already fetched. The CLI calls `POST /v1/snapshots//highlights`, so the page is not fetched again. ```bash expandai snapshots get 019c0000-0000-7000-8000-000000000020 --highlights "retry policy" ``` | Argument | Definition | | ------------ | -------------------------------------------------------------------------------------------------------- | | `snapshotId` | `meta.snapshotId` from an earlier `--format json` Fetch, or the Snapshot ID in the note a hook prepends. | | Option | Definition | Default | | --------------- | ---------------------------------------------------------- | ------- | | `--highlights` | Query for relevant passages in the Snapshot. | None | | `--format` | `text` prints the snippets. `json` prints the JSON object. | `text` | | `--max-results` | Most snippets to print, `1` to `50`. | `5` | | `--min-score` | Lowest relevance score a snippet needs, `0` to `1`. | `0.6` | Snapshot Highlights uses the Main Markdown and State JSON. The Appendix is not searched from the CLI. **Warning:** Pass `meta.snapshotId`, the capture's own ID. `meta.resources.snapshotId` is the public Snapshot ID and returns `404 FetchSnapshotHighlightsNotFound`. See [Two IDs](https://expand.ai/docs/wayback-machine#two-ids). ## Appendix ### `expandai appendix ` Read the [Appendix](https://expand.ai/docs/fetch#appendix) of a page you already fetched. Where Fetch moved navigation, a footer, or related links out of the Main Markdown, the Markdown text carries a marker on its own line: ```txt [appendix omitted: 14 links · ~320 tokens] {612} ``` `{612}` is the evidence ID of the first omitted block. Pass it with `--id` to read that block, or leave `--id` off for the whole Appendix. ```bash expandai appendix 019c0000-0000-7000-8000-000000000020 --id 612 expandai appendix https://expand.land/s/019c0000-0000-7000-8000-000000000020/ ``` | Argument | Definition | | ------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `snapshotId` | `meta.snapshotId` from a `--format json` Fetch, or the capture URL from the `playground:` frontmatter line of Markdown text. | | Option | Definition | Default | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | | `--id` | The evidence ID from a marker. | The whole Appendix | | `--format` | `text` prints a citation line, then the Markdown. `json` prints `snapshotId`, `evidenceId`, `playground`, `citationUrl`, and `markdown`. | `text` | The command reads the public Snapshot routes on the [Playground](https://expand.ai/docs/wayback-machine/playground-and-replay) host. It needs no API key and starts no new Fetch. **Note:** A bare ID is resolved against `EXPAND_PLAYGROUND_HOST`, which defaults to the production Playground. Pass the capture URL when the Snapshot came from another stack. ## Auth and configuration Signing in versus API keys, and how each surface sends the key, is on [Authentication](https://expand.ai/docs/authentication). ### `expandai login` Store credentials for the CLI, the local MCP server, and the hooks. ```bash expandai login ``` In a terminal the command asks how you want to authenticate. **Browser Login** opens a page where you approve the device. **API Key** prompts for a key and never shows it in your shell history. When credentials are already stored, the command asks before replacing them. When no terminal is attached, or a coding agent is running the command, only `expandai login --json` is accepted. It starts or resumes one device authorization and prints one JSON result. `status` is `authorization_required`, with `authorization.url` and `authorization.userCode` to show the user, `completed`, or `error`. Run the same command again after the user approves. ### `expandai whoami` Show the account, organization, and credential the CLI is using. ```bash expandai whoami expandai whoami --format json ``` | Option | Definition | Default | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | | `--format` | `text` prints a short status. `json` prints `authenticated`, `user`, `organization`, and `authentication`. When not signed in, `mode` is `preview`, `unauthenticated`, `credential-rejected`, or `identity-unavailable`. | `text` | With no stored credentials, `whoami` reports `Anonymous (free preview)`, and every Fetch runs against a free preview session until you sign in. ### `expandai logout` Clear stored credentials. ```bash expandai logout ``` In a terminal the command also offers to remove installed hooks, Skills, and MCP entries. Outside a terminal it only clears credentials. Run `expandai uninstall` for the rest. ### Environment and config | Name | Definition | | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EXPAND_API_KEY` | An API key. A stored login from `expandai login` or `expandai init` takes precedence over it, unless the login was issued by a different API origin. The CLI logs a warning either way. | | `EXPAND_API_URL` | The API origin. Overrides the config file. | | `EXPAND_APP_URL` | The web app origin the CLI points agents at. Default `https://expand.ai`. | | `EXPAND_PLAYGROUND_HOST` | The Playground host used to resolve bare Snapshot IDs and build citation links. | | `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY` | Standard proxy variables. Upper or lower case. | | `XDG_CONFIG_HOME` | Where the config file lives. | | `EXPANDAI_CLI_CONTEXT` | `human` or `agent`. Forces the command set described under [Agent setup](https://expand.ai/docs/cli-commands#agent-setup). | | `EXPANDAI_NO_TELEMETRY` | Set to `1` to turn off anonymous usage telemetry. `DO_NOT_TRACK` works too. | | `DEBUG` | Set to `true` for debug logging on stderr. | Credentials and settings are stored in `$XDG_CONFIG_HOME/expandai/config.json`, which is `~/.config/expandai/config.json` on every platform when `XDG_CONFIG_HOME` is unset. ## Agent setup These commands wire Expand into a coding agent. The hosted MCP server needs none of them. See [Connect your agent (MCP)](https://expand.ai/docs/mcp-server) for that path, and [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks) for what the local setup adds. Every install takes a `` and a ``. Hosts are `claude-code`, `cursor`, `opencode`, and `codex`. Scope is `global`, for every session on the machine, or `project`, which writes into the current working directory. Run project installs from the project root. **Note:** The CLI shows a smaller command set when a coding agent runs it. When stdout is not a terminal, or `CLAUDECODE`, `CURSOR_TRACE_ID`, or `CODEX_RUN_ID` is set, `mcp install`, `mcp uninstall`, `skill`, `hook install`, and `hook uninstall` are hidden, and `expandai init --json` is the setup command. `EXPANDAI_CLI_CONTEXT=human` or `=agent` overrides the detection. ### `expandai init` Sign in and configure the coding agents on this machine in one step. ```bash expandai init expandai init --detect expandai init --harnesses claude-code,cursor --scope global ``` | Option | Definition | Default | | --------------- | ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------- | | `--detect` | Print the agents setup would configure, and change nothing. | Off | | `--harnesses` | Comma-separated agents to configure, from `claude-code`, `cursor`, `opencode`, `codex`. | Every detected agent | | `--harness` | The agent running the command, as a detection hint. | None | | `--scope` | `project` or `global`. | Asked in a terminal. Required with `--json`. | | `--setup-token` | A short-lived, single-use token from the dashboard's **Copy setup prompt**. Authorizes setup with no browser step. | None | | `--json` | Print one JSON result. Required when an agent runs the command. | Off | In a terminal, `init` asks for the scope, shows the detected agents with all of them selected, and, when no usable credentials are stored, prints a URL and a code to approve in the browser. What each agent receives: | Agent | What `init` configures | | ----------- | --------------------------------------------------------------------------- | | Claude Code | The Expand Skill and the MCP server. Nothing is written to `settings.json`. | | Cursor | The Expand Skill and the MCP server. | | Codex | The Expand Skill and the MCP server. | | OpenCode | The Expand Skill, the MCP server, and the web-fetch hook. | With `--json`, a sign-in that needs the browser returns `status: "authorization_required"` with `authorization.url` and `authorization.userCode`. Run the same command again after the user approves. Nothing is pasted back. ### `expandai status` Report what is configured for one agent and scope. The command reads and writes nothing. ```bash expandai status --harness claude-code --scope global expandai status --harness opencode --scope project --json ``` | Option | Definition | Default | | ----------- | ------------------------------------------------ | -------------------------------------------- | | `--harness` | `claude-code`, `cursor`, `opencode`, or `codex`. | Asked in a terminal. Required with `--json`. | | `--scope` | `project` or `global`. | Asked in a terminal. Required with `--json`. | | `--json` | Print one JSON result. | Off | The text output lists the auth state, any pending authorization, whether the Skill and the MCP server are configured, the web-fetch hook for OpenCode, and whether the agent needs a restart to pick the configuration up. ### `expandai mcp run` Run the Expand MCP server over stdio. ```bash expandai mcp run expandai mcp run --docs ``` | Option | Definition | Default | | -------- | --------------------------------------------------------------- | ------- | | `--docs` | Also serve the `expand://about` and `expand://start` resources. | Off | Agents launch this command from their MCP config. You rarely run it by hand. It uses the credentials from `expandai login`, so no key is written into the agent's config. The tools are on [MCP tools & resources](https://expand.ai/docs/mcp-tools-and-resources). ### `expandai mcp install ` Register the MCP server in one agent's config. `expandai mcp uninstall ` removes it. ```bash expandai mcp install cursor project expandai mcp uninstall cursor project ``` | Host | Global config | Project config | | ------------- | ---------------------------------- | -------------------- | | `claude-code` | `~/.claude.json` | `.mcp.json` | | `cursor` | `~/.cursor/mcp.json` | `.cursor/mcp.json` | | `opencode` | `~/.config/opencode/opencode.json` | `opencode.json` | | `codex` | `~/.codex/config.toml` | `.codex/config.toml` | The entry is named `expandai` and runs `expandai mcp run`. When `expandai` is not on `PATH` at install time, the entry runs `npx -y expandai mcp run` instead. ### `expandai skill install ` Copy the Expand Skill into one agent's skills directory. `expandai skill uninstall ` removes it. ```bash expandai skill install claude-code project expandai skill uninstall claude-code project ``` | Host | Global directory | Project directory | | ------------- | ------------------------------------- | ---------------------------- | | `claude-code` | `~/.claude/skills/expandai/` | `.claude/skills/expandai/` | | `cursor` | `~/.cursor/skills/expandai/` | `.cursor/skills/expandai/` | | `opencode` | `~/.config/opencode/skills/expandai/` | `.opencode/skills/expandai/` | | `codex` | `~/.agents/skills/expandai/` | `.agents/skills/expandai/` | The Skill is one `SKILL.md`. What it teaches is on [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks#the-expand-skill). **Recommendation:** Commit a project-scoped Skill only when you want it enabled for everyone who clones the repository. ### `expandai hook install ` Install the Expand Skill, register the MCP server, and install the web-fetch hook for one agent, in that order. Hosts are `claude-code` and `opencode`. `expandai hook uninstall ` removes the hook and the MCP entry. ```bash expandai hook install opencode project expandai hook uninstall opencode project ``` | Host | Hook | | ------------- | --------------------------------------------------------------------------------------------------------- | | `opencode` | A plugin file, `expandai-webfetch.js`, in `~/.config/opencode/plugins/` or `.opencode/plugins/`. | | `claude-code` | None. The hook step prints a notice and writes nothing. The Skill and the MCP server are still installed. | What the hook does is on [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks#what-the-hook-does). ### `expandai hook claude-code` and `expandai hook opencode` The protocol runners the hooks call. Each reads one JSON payload from stdin, fetches the URL in it through Expand, and writes the result to stdout. ```bash expandai hook opencode ``` Agents run these when a hook fires. You do not call them yourself. ## Cleanup ### `expandai uninstall [target] [host] [scope]` Remove hooks, Skills, and MCP entries. ```bash expandai uninstall expandai uninstall mcp codex project expandai uninstall skill claude-code global ``` | Argument | Definition | Default | | -------- | ------------------------------------------------ | ----------- | | `target` | `all`, `hook`, `skill`, or `mcp`. | `all` | | `host` | `claude-code`, `cursor`, `opencode`, or `codex`. | Every host | | `scope` | `global` or `project`. | Both scopes | **Warning:** Project-scoped files are looked up from the current working directory. Run `uninstall` from the same project root you installed from, or the files are not found. ## Telemetry The CLI sends anonymous usage events, and says so the first time you run it in a terminal. ```bash expandai telemetry status expandai telemetry disable expandai telemetry enable ``` `EXPANDAI_NO_TELEMETRY=1` or `DO_NOT_TRACK=1` turns it off without changing the config file. ## Crawl `expandai crawl ` crawls a site into a folder of Markdown files you can resume. `expandai sync [url] [directory]` creates or upgrades a synchronized crawl and keeps the folder current. Its subcommands are `sync status`, `sync run`, `sync install`, which installs a systemd user timer on Linux or a launchd agent on macOS, and `sync uninstall`. ```bash expandai crawl https://docs.stripe.com ./stripe-docs --include '/docs/**' expandai sync https://docs.stripe.com ./stripe-docs --every 6h ``` Both commands exit `0` when nothing changed, `3` when at least one page changed, `2` on a usage error, and `1` on any other failure. `expandai crawl --help` and `expandai sync --help` list every flag. What a crawl is, and how it finds pages, is on [Crawl](https://expand.ai/docs/crawl) and [WebSync](https://expand.ai/docs/websync). ## Next steps - [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks): what the local setup adds, and what the Skill teaches an agent. - [MCP tools & resources](https://expand.ai/docs/mcp-tools-and-resources): the tools behind `expandai mcp run`. - [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight): Inline and Snapshot Highlights, and how to tune them. - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url): the two modes and the include options the CLI leaves at their defaults. --- # CLI, Skill & hooks (https://expand.ai/docs/cli-skill-and-hooks) What installing the Expand CLI adds on top of the hosted MCP server. Terminal fetches, the Expand Skill, the OpenCode web-fetch hook, and proxy support. The hosted [MCP server](https://expand.ai/docs/mcp-server) gives an agent Fetch with nothing installed. The `expandai` CLI is the local path. Install it when you want one of these: - **Fetch from a terminal.** `expandai fetch ` prints Markdown text, for you or for an agent that runs shell commands. - **The Expand Skill.** A `SKILL.md` the agent loads on demand. It says when to use Expand, how to cite what it fetched, and how to read the Appendix. - **Reroute OpenCode's own web fetch.** A hook sends every `webfetch` call through Expand, so the agent gets rendered Markdown without being told to use a specific tool. - **A restricted network.** The CLI honors `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`. ## Install and sign in ```bash npm install -g expandai expandai login ``` `expandai login` offers browser sign-in or an API key prompt. Credentials are stored in `~/.config/expandai/config.json`, or under `$XDG_CONFIG_HOME` when it is set. No key is written into any agent's config file. Every command also runs without a global install, through `npx -y expandai `. ## One command per agent `expandai init` finds the coding agents on your machine, signs you in if needed, and configures the ones you pick. ```bash expandai init --detect expandai init ``` `--detect` lists what would be configured and changes nothing. Without it, `init` asks for a scope, shows the detected agents with all of them selected, and writes to the ones you confirm. Every agent gets the Expand Skill and the MCP server, and OpenCode also gets the web-fetch hook. The per-agent table is on [CLI: Commands](https://expand.ai/docs/cli-commands#expandai-init). Scope is `global` or `project`. Project scope writes into the current repository, so collaborators who clone it inherit the setup. Global scope applies to every session on the machine. **Note:** `init` does not write to Claude Code's `settings.json`, does not change its `permissions`, and does not install a hook there. Claude Code's built-in `WebFetch` keeps working as before, and the Expand Skill is what routes fetches through Expand. To configure one agent and skip the questions, pass `--harnesses` and `--scope`: ```bash expandai init --harnesses codex --scope project ``` Every flag, and the per-host `mcp install`, `skill install`, and `hook install` commands, are on [CLI: Commands](https://expand.ai/docs/cli-commands#agent-setup). ## The Expand Skill The Skill is one Markdown file, `SKILL.md`, named `expandai`. `init`, `skill install`, and `hook install` copy it into the agent's skills directory, such as `~/.claude/skills/expandai/` for Claude Code or `.agents/skills/expandai/` for a Codex project. Every agent gets the same file. It tells the agent: - To run `expandai fetch ` before `curl` or a built-in web-fetch tool on any public page, and to skip Expand for localhost, private hosts, and JSON APIs. - To run `expandai snapshots get --highlights ` when it needs passages from a page it already fetched, instead of fetching the URL again. - What an `[appendix omitted: …] {N}` marker is, that it is not page content, and to read the block with `expandai appendix` rather than refetching. - How to follow the `more` link in the frontmatter to the Snapshot's catalog. See [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). - To cite every claim it takes from a page with a `**Sources**` block of `?id=` links, and how to narrow a table citation to one cell. See [Citation links](https://expand.ai/docs/wayback-machine/playground-and-replay#citation-links). - To recover from an authentication error by running `expandai login --json` itself and relaying the URL and code to the user. **Note:** A new CLI version ships a new `SKILL.md`. `init` and `status` compare the installed file with the bundled one, so an outdated Skill shows as not configured until you run `init` again. ## What the hook does OpenCode's `webfetch` tool has no JavaScript engine, so a client-rendered page comes back as an empty shell. The hook is a plugin file, `expandai-webfetch.js`, that `init` or `hook install` copies into `~/.config/opencode/plugins/` or `.opencode/plugins/`. After every `webfetch` call it runs `expandai hook opencode` with the URL, and replaces the tool's output with the Markdown text Expand returns. **Note:** This is the one thing the hosted MCP server cannot do. A remote server can offer tools, but it cannot change what the agent's own tools return. The Markdown starts with a short note that carries the Snapshot ID and the capture link, and tells the agent to call `get_snapshot` or `expandai snapshots get` for more passages rather than fetching the URL again. **Note:** Claude Code gets no hook. `expandai hook install claude-code` installs the Skill and the MCP server, then prints a notice for the hook step and writes nothing. The protocol runner `expandai hook claude-code` still exists, but setup does not register it. ## Verifying and removing ```bash expandai status --harness opencode --scope project expandai whoami expandai uninstall ``` `status` reports whether the Skill, the MCP server, and, for OpenCode, the hook are configured, and whether the agent needs a restart. It changes nothing. `uninstall` with no arguments removes every hook, Skill, and MCP entry the CLI installed, in both scopes. **Warning:** Agents read their configuration when they start. After `init`, restart the agent, including the one that ran the command. ## Running MCP locally `expandai mcp run` serves the same three tools as the hosted server, over stdio, using the credentials from `expandai login`. `init` and `mcp install` register it for you. Use it where an editor cannot reach `https://api.expand.ai`. The config entry and the tools are on [MCP tools & resources](https://expand.ai/docs/mcp-tools-and-resources#local-over-stdio). ## Windows Everything above works without an elevated shell. Credentials are stored under your user profile. The CLI tightens the file's access control list when it can. When Windows refuses, which it does for a non-elevated account, the CLI logs a warning and continues. Run one elevated shell if you want the list tightened. ## Next steps - [Connect your agent (MCP)](https://expand.ai/docs/mcp-server): the hosted server, with no local process. - [CLI: Commands](https://expand.ai/docs/cli-commands): every command, flag, and default. - [MCP tools & resources](https://expand.ai/docs/mcp-tools-and-resources): the tools an agent gets, with their parameters and results. - [Docs for agents](https://expand.ai/docs/docs-for-agents): let an agent set Expand up itself from `start.md`. --- # Docs for agents (https://expand.ai/docs/docs-for-agents) Let an agent set Expand up from start.md, and give an agent Expand's docs as Markdown, one page or all of it. Every Expand doc is readable by an agent. Hand it the file for the job: a setup guide, an index, the whole docs bundle, or one page as Markdown. ## Which one should I use? | Need | Use | | ------------------------------------------ | ------------------------------------------------------------- | | Set up Expand in a coding agent | [`/start.md`](https://expand.ai/start.md) | | First orientation | [`/llms.txt`](https://expand.ai/llms.txt) | | The docs table of contents | [`/docs/llms.txt`](https://expand.ai/docs/llms.txt) | | Every docs page, in one pull | [`/docs/llms-full.txt`](https://expand.ai/docs/llms-full.txt) | | One docs page | The page's URL with `.md` added | | One API reference page, with exact schemas | The page's URL with `.md` added, under `/api-reference/` | **Note:** These files are for agents and tools you point at Expand's docs: coding agents, IDE assistants, and MCP clients. They are not a signal to AI search engines. ## start.md `https://expand.ai/start.md` is the setup guide for coding agents. It holds the whole procedure in one document, served as `text/markdown` and never cached, so an agent fetches it once and starts. Paste this into any coding agent: ```txt Read https://expand.ai/start.md and set up Expand. Fetch it as raw text with curl. A summarizing web-fetch tool drops the exact commands. ``` The same prompt is at the end of `/llms.txt`. The **Copy setup prompt** button in the app produces it too, with a fresh single-use setup token attached, so the agent can authorize setup without a browser step. **Warning:** The second sentence matters. Some web-fetch tools return a summary of a page instead of its text, and a summary of a procedure loses the commands. What the guide walks the agent through: | Section | What the agent does | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Fastest path | Adds the hosted MCP server, `https://api.expand.ai/v1/mcp`, when the user only wants fetches. Then stops. | | Setup | Names the agent it is running in, runs `expandai init --detect --json`, asks which agents to configure, and runs one `expandai init` with `--harnesses`, `--scope`, and `--json`. | | Authorization | Passes the setup token when the prompt carried one. Otherwise relays the browser URL and code from `authorization_required`, and runs the same command again once the user approves. | | Confirm Expand works | Runs one `expandai fetch --highlights` against a live page and answers with the exact captured sentence and its citation link. | | Next steps | Says which agents to restart, offers a first fetch, and offers to set up the TypeScript SDK when the conversation is inside a project. | | Beyond agent setup | The SDK, the HTTP API, and the CLI, for callers that are not coding agents. | The exact commands live in the guide. `init` configures only the agents the user picks and leaves the rest alone. What each agent receives is on [CLI: Commands](https://expand.ai/docs/cli-commands#expandai-init). Use the guide whenever you want Expand set up in a coding agent. Use [Connect your agent (MCP)](https://expand.ai/docs/mcp-server) when you would rather add the hosted server by hand. **Note:** `expand://start` is an MCP resource that points at `/start.md` and carries the same prompt. The hosted MCP server always serves it. A local `expandai mcp run` serves it only with `--docs`. See [MCP tools & resources](https://expand.ai/docs/mcp-tools-and-resources#documentation-resources). ## llms.txt `/llms.txt` is a short, hand-written entry point. It links the start guide, the docs, the API reference, and the full docs bundle, and ends with the setup prompt above. Give it to an agent that needs to find its way around before it does anything. `/docs/llms.txt` is a different file. It is generated from the docs and lists every docs page as a Markdown link with the page's title and one-line description. Give it to an agent that needs a table of contents. It does not list API reference pages. ## llms-full.txt `/docs/llms-full.txt` is every docs page as Markdown, one after another. Each page opens with a heading that carries its title and canonical URL, then the page body, then a `---` rule before the next page, so an agent can cite or revisit the source page. ```md # Quickstart (https://expand.ai/docs/quickstart) ... --- # Fetching a URL (https://expand.ai/docs/fetch/fetching-a-url) ... ``` Use it when the agent needs the docs as a whole and has the context budget for them. It does not contain the API reference, and it can be too large for a small context window. **Recommendation:** For one topic, give the agent that page's `.md` URL instead of the bundle. ## Any page as Markdown Every docs page and every API reference page is available as Markdown. Append `.md` to the URL. | Page | Markdown | | -------------------------------------- | ----------------------------------------- | | `/docs/quickstart` | `/docs/quickstart.md` | | `/api-reference/fetch/fetch.fetchJson` | `/api-reference/fetch/fetch.fetchJson.md` | ```bash curl https://expand.ai/docs/quickstart.md ``` The response is `text/markdown`. It opens with the page title and its canonical URL, then the content with code blocks, tables, links, and all tab examples in plain Markdown. Interactive examples have static instructions in the Markdown export. API endpoint pages include request and response schemas from the OpenAPI document. `.mdx` in the URL returns the same thing. ```md # Quickstart (https://expand.ai/docs/quickstart) ... ``` A tool that can set request headers can ask the human URL for Markdown instead, with `Accept: text/markdown`. This works for `/docs` and `/api-reference` pages, not for every route on the site. ```bash curl -H "Accept: text/markdown" https://expand.ai/docs/quickstart ``` On every docs page, the **Copy as Markdown** button copies the same Markdown the `.md` URL serves, and its menu opens the page in ChatGPT or Claude, or shows the Markdown itself. ### Giving pages to an agent One `.md` page when the agent needs help with one feature. Two or three when the task spans a workflow. The full bundle only when it needs everything. ```txt Read these Expand docs before answering: - https://expand.ai/docs/fetch/fetching-a-url.md - https://expand.ai/docs/highlights/two-ways-to-highlight.md Then explain how to build a research agent whose answers carry citation links. ``` The docs explain how to use Expand. The API reference holds the exact request and response schemas, parameters, and errors. When the agent needs that precision, give it the API reference page's `.md` URL. ## Guidance for agents - Prefer the current Markdown docs over memorized behavior. - Use the API reference for exact request and response shapes. - Read one `.md` page before pulling the full bundle. - Do not assume an endpoint, CLI flag, SDK method, or schema exists unless the docs show it. - Cite and revisit a page by the canonical URL in its Markdown heading. ## Next steps - [Connect your agent (MCP)](https://expand.ai/docs/mcp-server): add the hosted MCP server to a coding agent by hand. - [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks): what `start.md` installs, and what the Skill teaches. - [Quickstart](https://expand.ai/docs/quickstart): run one Fetch. - [API reference](https://expand.ai/api-reference): the exact endpoint schemas, parameters, and errors. --- # Errors & retries (https://expand.ai/docs/errors-and-retries) How to tell an Expand failure from a dead URL, which errors to retry, and what you are charged for. Check the HTTP status and the error body together: **A `200` can still report an origin failure. A non-2xx can be an API or capture error, or an origin refusing to serve a file.** Browser pages and files report origin errors differently below. Nothing retries a dead URL by accident. Every response, success or error, carries an `x-expand-trace-id` header. Include it when you contact support. This page is the guide. Every status code, error type, and example is in [API reference: Errors](https://expand.ai/api-reference/errors). ## The two error shapes **When a request fails with a non-2xx response**, the body carries a `_tag` that names the error and, for errors that might be temporary, a `retryable` boolean: ```json { "_tag": "ServiceUnavailable", "message": "…", "retryable": true } ``` Errors that are unambiguous from their status alone (`400`, `401`, `403`, `404`, `409`, `410`, `413`) omit `retryable`. Treat a missing field on those as `false`. **When a browser page's origin fails**, Fetch returns `200` with a `FetchOriginError` instead of, or next to, the page. A dead page URL, a broken certificate, or a site that answers `404` is a `200` with an origin error attached. A [file's origin](https://expand.ai/docs/fetch/documents#document-errors) can instead return `404`, `410`, or `502` with `FetchOriginHttpError`. The fields, the headers on `POST /v1/fetch`, and examples are in [When the origin fails](https://expand.ai/docs/errors-and-retries#when-the-origin-fails). **Crawl** uses its own shape: `{ "error": { "code", "message", "retryable" } }`. See [Crawl errors](https://expand.ai/docs/crawl#crawl-errors). ## When the origin fails Not every URL can be fetched. The domain may not resolve, the certificate may be broken, nothing may be listening, or the origin may answer with a `404`. Those are the origin's failures, not Expand's, and Fetch reports them as such. `POST /v1/fetch/json` returns a typed `FetchOriginError` instead of a document when the origin was never reached: ```json { "_tag": "FetchOriginError", "url": "https://checkout.deadstartup.example/pricing", "kind": "dns", "upstreamError": "net::ERR_NAME_NOT_RESOLVED", "retryable": false } ``` When a browser page's origin *did* answer with an error status, the captured content still comes back with the outcome attached, because an error page is often the answer you needed: ```json { "meta": { "url": "https://site.example/removed-page", "title": "Page not found" }, "markdown": "# Page not found\n\n…", "json": [], "originError": { "_tag": "FetchOriginError", "url": "https://site.example/removed-page", "kind": "http", "originStatusCode": 404, "retryable": false } } ``` `POST /v1/fetch` answers in Markdown, so the same facts ride the `x-expand-origin-error`, `x-expand-origin-error-detail`, and `x-expand-origin-error-retryable` response headers. When no document exists, the body is a short Markdown error document an agent can read directly. A file has no error page to render. If its origin refuses it, Fetch returns a non-2xx `FetchOriginHttpError`: `404` or `410` for a missing or removed file, and `502` for other origin statuses. See [Document errors](https://expand.ai/docs/fetch/documents#document-errors). For browser origin outcomes, `kind` says what went wrong and `retryable` says whether trying again could help. Network-error verdicts use the exact Chromium code; HTTP-error verdicts use the origin status. Every `kind`, field, and header is in [API reference: Errors](https://expand.ai/api-reference/errors#origin-outcomes-200). ## Should I retry? Trust `retryable`: | Outcome | Retry? | | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `200` with `FetchOriginError`, `retryable: true` | Yes, after a delay. The condition may clear. | | `200` with `FetchOriginError`, `retryable: false` | No. The URL is dead. Fix the URL or drop it. | | `200` with no origin error | Nothing to retry. | | `404` or `410` with `FetchOriginHttpError` for a file | No. The file is missing or gone. | | `400`, `401`, `403`, `413` | No. Fix the request, the credentials, the balance, or the URL. | | `429` | Yes, following `retryable` and `Retry-After`. See [Limits](https://expand.ai/docs/limits#concurrency-and-rate). | | `500` | Yes when the body says `retryable: true`. Report it with the trace ID if it persists. | | `502` with `FetchNavigationFailed` or `FetchCaptureFailed` | No. The page or the capture fails the same way every time, so another attempt reaches the same result. | | `502` with `FetchOriginHttpError` for a file | Yes. The file's origin may serve it later; the SDK retries this outcome. | | `503` | Yes, after `Retry-After`. Expand could not prove the origin or the page was at fault, so the failure counts as Expand's and may clear. | | `504` with `FetchCaptureTimeout` | No automatic retry. A deliberate retry may still be appropriate. | | `529` with `FetchCapacityTimeout` | Yes, after `Retry-After`. The capture never started. | **Warning:** Do not treat a `200` carrying an origin error as a transport failure. It is a completed answer about a URL that could not serve content. When a response carries a `Retry-After` header (`429`, `503`, and `529` do), wait that many seconds. Otherwise back off exponentially with jitter, up to about 32 seconds. Hitting a limit is the most common reason for a `429`. What the limits are is on [Limits](https://expand.ai/docs/limits). ## The SDK retries for you The TypeScript SDK follows the same rules. It retries when an error says `retryable: true`, up to `maxRetries` (default `2`) within one `timeoutMs` budget (default `60000`), which covers the retries and the waits between them. It never retries the Batched Fetch `409` conflicts, the `502` page errors, or the `504` capture timeout, because repeating them reaches the same answer. It does retry `529 FetchCapacityTimeout` and `503 FetchDocumentProcessingUnavailable` with normal backoff, and `502 FetchOriginHttpError` for a document. It does not retry `403`, `413`, `415`, a `404` or `410` document refusal, or a `200` origin outcome. Older TypeScript SDK versions expose an unrecognized `404` or `410` body as raw JSON text. Upgrade the SDK if you read structured fields from the error body. Crawl calls surface errors as a `CrawlClientApiError` with `code`, `status`, and `retryable`. Options are in [TypeScript SDK: Errors and retries](https://expand.ai/docs/typescript-sdk#errors-and-retries). ## What you are charged for You pay for delivered content, not for attempts. Charges follow the outcome, not the status code. | Outcome | Billed | | ----------------------------------------------------------------------- | -------------------------------------------------------------- | | Page captured successfully | Yes, one credit | | Origin responded with an error status (`400`+) and content was captured | **Yes**, the content was delivered alongside the `originError` | | Origin unreachable, any network `FetchOriginError` kind | No | | `FetchBlocked` (403) | No | | `FetchPageTooLarge` (413) | No | | `ServiceUnavailable` (503) or `InternalError` (500) | No | The one case that surprises people is the second row: an origin's `404` or `500` page still contains content, such as a soft-404, a maintenance page, or a status dashboard, and Expand captured and returned it, so it costs a credit like any other page. If you do not want to pay for error pages, check `originError` before treating a response as useful. Everything else is straightforward: if no content reached you, no charge was made. See [Pricing & Usage](https://expand.ai/docs/pricing-and-usage#what-you-are-charged-for-by-outcome) for what each operation costs in credits. **Note:** A request made with no credits left is refused before any work starts. ## Handle errors in code ### With the TypeScript SDK `ExpandClient` throws for non-2xx API responses, including a file origin's `FetchOriginHttpError`. Inspect `status` and `body` on `ExpandClientApiError` to branch on the `_tag`. See the [TypeScript SDK](https://expand.ai/docs/typescript-sdk) for the full error hierarchy. ```typescript import { ExpandClient, ExpandClientApiError, ExpandClientError } from '@expandai/sdk' const expand = new ExpandClient() try { const result = await expand.fetchJson({ url: 'https://example.com' }) // Reached here means the API returned 200. Check result for an origin outcome. } catch (error) { if (error instanceof ExpandClientApiError) { // A non-2xx response. `body` is the parsed error above. Branch on `_tag`. console.log('Status:', error.status) console.log('Tag:', (error.body as { _tag?: string })?._tag) } else if (error instanceof ExpandClientError) { // Everything else the SDK can throw: timeouts, connection failures, bad options. console.log('Error:', error.message) } } ``` The SDK already retries for you, using the same `retryable` field. It never retries the `502` page errors, `FetchNavigationFailed` and `FetchCaptureFailed`, and it waits out `Retry-After` before retrying a `429`. Anything that reaches your `catch` block has run out of attempts. `fetchJson` resolves to either shape, so narrow before reading content. Only the origin-outcome variant carries a top-level `_tag`: ```typescript import { isFetchOriginError } from '@expandai/sdk' const result = await expand.fetchJson({ url: 'https://example.com' }) if (isFetchOriginError(result)) { // The origin was never reached, so there is no content. console.log(result.kind, result.retryable) } else if (result.originError) { // The origin answered with an error status, but the page was captured. console.log(result.originError.originStatusCode, result.markdown) } else { console.log(result.markdown) } ``` ### With cURL Check the status code first, then the body. On `/v1/fetch`, the origin outcome is in the response headers. ```bash response=$(curl -s -D /tmp/expand-headers -w "\n%{http_code}" \ -X POST https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}') status_code=$(echo "$response" | tail -n 1) body=$(echo "$response" | sed '$d') if [ "$status_code" -ne 200 ]; then # API, capture, or file-origin error. The body carries a _tag. echo "Fetch error ($status_code): $body" elif grep -qi '^x-expand-origin-error:' /tmp/expand-headers; then # Expand succeeded; the origin did not. grep -i '^x-expand-origin-error' /tmp/expand-headers fi ``` ### Without an SDK If you are calling the API directly rather than through an SDK, retry only when the body says to, and never sooner than `Retry-After` asks: 1. Read `retryable` from the error body. If it is `false`, stop. No retry can succeed 2. Wait at least `Retry-After` seconds (or the body's `retryAfterSeconds`, if present) 3. Back off exponentially from there, up to a maximum (e.g., 32 seconds) 4. Jitter the wait. Every client rejected in the same burst is handed the same `Retry-After`, so an unjittered wait sends the whole population back at one instant 5. After your maximum attempts, surface the error to the user A successful `/v1/fetch` responds with `text/markdown`, not JSON. Read it with `response.text()`. Only the error responses are JSON, and a failure from an intermediary (a proxy's own error page, an empty body) may not be JSON either, so parsing it has to be allowed to fail. ```typescript async function fetchWithRetry(url: string, maxRetries = 5) { let delay = 1000 for (let attempt = 0; attempt < maxRetries; attempt++) { const response = await fetch('https://api.expand.ai/v1/fetch', { method: 'POST', headers: { 'x-expand-api-key': API_KEY, 'content-type': 'application/json' }, body: JSON.stringify({ url }), }) // The success body is the Markdown document itself. if (response.ok) return response.text() // An undecodable error body tells you nothing about retryability. Fall back to // the status: 408/429/5xx are the transient ones. 409 is a conflict verdict, // not a transient failure. Retrying reaches the same answer. const body = await response.json().catch(() => undefined) const retryable = typeof body?.retryable === 'boolean' ? body.retryable : response.status === 408 || response.status === 429 || response.status >= 500 if (!retryable || attempt === maxRetries - 1) throw new Error(body?._tag ?? `HTTP ${response.status}`) // The body wins over the header: it survives proxies that strip headers, and it is the // value the server chose for this error rather than the default for this status. const retryAfter = Number(body?.retryAfterSeconds ?? response.headers.get('retry-after') ?? 0) * 1000 const wait = Math.max(delay, retryAfter) // Spread over [1x, 1.25x], upward only, since `Retry-After` names the earliest // moment worth trying again and jittering below it retries sooner than asked. await new Promise(resolve => setTimeout(resolve, wait * (1 + Math.random() * 0.25))) delay = Math.min(delay * 2, 32_000) } } ``` ## Best practices 1. **Check the status and the body.** A `200` may carry an origin outcome. A non-2xx may be an API or capture error, or a file origin's `FetchOriginHttpError` refusal (`404`, `410`, or `502`). 2. **Branch on `_tag`,** not on message text. 3. **Trust `retryable` when present.** For browser origin network errors, it is derived from the exact Chromium code. `false` means stop, and `true` means the condition may clear. For a file-origin `FetchOriginHttpError`, use the HTTP status because its body has no `retryable` field. 4. **Retry transient outcomes** with backoff and never sooner than `Retry-After`. The SDK also retries a file-origin `502 FetchOriginHttpError`. 5. **Do not automatically retry** page verdicts (`403`, `413`, `502`) or the conservative timeout fallback (`504`). A deliberate 504 retry may still be appropriate, but each attempt re-runs a full capture. 6. **Allow time for the content you fetch:** use a `180000` ms client timeout for browser captures or `3600000` ms for documents. Documents can process for up to 55 minutes. 7. **Record `x-expand-trace-id`** on every response. It is what support needs, and it is present even when the body is not. 8. **Validate inputs** before sending to avoid `400` errors. ## Next steps - [API reference: Errors](https://expand.ai/api-reference/errors): every status code, error type, and example. - [Limits](https://expand.ai/docs/limits): concurrency, timeouts, and sizes. - [Pricing & Usage](https://expand.ai/docs/pricing-and-usage#what-you-are-charged-for-by-outcome): what each outcome costs. --- # FAQ (https://expand.ai/docs/faq) Short answers to common questions about Fetch, agents, Highlights, billing, tiers, and the docs themselves, each with a link to the page that has the detail. Short answers to common questions. Each one links to the page that has the detail. ## Using Expand ### What is Fetch? Fetch is Expand's core product. You send a URL and get the page back as Markdown text or a JSON object, with the [State JSON](https://expand.ai/docs/fetch#state-json) behind it and a [Snapshot](https://expand.ai/docs/fetch#snapshot--evidence) every line links back to. Start with the [Fetch overview](https://expand.ai/docs/fetch). ### Are the CLI, MCP, the TypeScript SDK, and the API separate products? No. They are ways to call Fetch. Use the CLI from a terminal, the TypeScript SDK or the API from application code, MCP from an agent, and hooks or the Skill to route an agent's own web tool through Expand. See [Quickstart: Ways to use Expand](https://expand.ai/docs/quickstart#ways-to-use-expand). ### Which one should I use? | Need | Use | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | Try one URL from your terminal | [Quickstart](https://expand.ai/docs/quickstart) with the CLI | | Add Fetch to an app | [TypeScript SDK](https://expand.ai/docs/typescript-sdk) or the [API reference](https://expand.ai/api-reference) | | Give a coding agent web context | [Connect your agent (MCP)](https://expand.ai/docs/mcp-server) | | Check what a Fetch returned, visually | [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay) | ### When should I use the CLI instead of the TypeScript SDK? Use the CLI to inspect a page locally, in scripts, and in an agent's shell. Use the TypeScript SDK or the API when Fetch is part of your application code. See [CLI: Commands](https://expand.ai/docs/cli-commands) and [TypeScript SDK](https://expand.ai/docs/typescript-sdk). ## Fetch output ### Why does `/v1/fetch` return Markdown text? `/v1/fetch` is Markdown Mode, for the common case where the page goes straight into a prompt. Use `/v1/fetch/json`, JSON Mode, when your code reads the fields separately, such as the Snapshot ID or the State JSON. See [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#two-modes). ### When should I use JSON Mode? Use JSON Mode when application code needs the Main Markdown, the State JSON, the Snapshot ID, or Fetch Highlights as separate fields of one JSON object. See [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#two-modes) and [TypeScript SDK](https://expand.ai/docs/typescript-sdk). ### What are the Main Markdown and the State JSON? The [Main Markdown](https://expand.ai/docs/fetch#main-markdown) is the page's content as Markdown, with navigation, footers, and ads taken out. The [State JSON](https://expand.ai/docs/fetch#state-json) is the data behind the page, such as the JSON its own APIs returned. See [How Fetch works](https://expand.ai/docs/fetch#how-fetch-works). ### Is the State JSON always included? The State JSON is on by default in both modes. An empty `?include=` turns it off. An entry too large to send comes back without `body`, with `moreTokens` and a sample of its `keys` instead, so you can ask for it when you need it. See [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#include-options). ### What is progressive disclosure? Fetch returns the Main Markdown first and keeps a pointer to everything it held back, such as the Appendix and large State JSON entries, so your agent reads more only when the task needs it. See [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). ### Are the Appendix pointer, the Snapshot ID, and `moreTokens` missing data? No. They are pointers to content Fetch held back to keep the response small. The Appendix, the replay, and the rest of a large State JSON entry are each one request away. See [References to more](https://expand.ai/docs/fetch/fetching-a-url#references-to-more). ### Should I turn on every include option? No. Start with the defaults. Add the Appendix, a screenshot, or other extras only when the task needs them. See [Include options](https://expand.ai/docs/fetch/fetching-a-url#include-options). ### Does Batched Fetch return everything a single Fetch does? No. Batched Fetch returns the Main Markdown and the State JSON, and does not return screenshots, the Appendix, Highlights, or converted files. See [Supported options](https://expand.ai/docs/fetch/batched-fetch#supported-options). ## Highlights and citations ### What is Highlights? Highlights returns the snippets of a page that answer a question, ranked by relevance, so your agent reads the passages it needs instead of the whole page. See [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight). ### Does Highlights fetch the page again? Not when you highlight a Snapshot you already have. Snapshot Highlights takes the `meta.snapshotId` from an earlier Fetch and costs `1` credit. Fetch Highlights runs as part of a new Fetch. See [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight). ### Can Highlights search the Main Markdown, the State JSON, and the Appendix? Yes. By default Highlights searches the Main Markdown and the State JSON. The Appendix is searched only when you include it. See [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight). ### Do Highlights snippets come with citations? Every snippet carries a `location` with an evidence ID. From MCP, each snippet also carries a `citationUrl`. From the API or the TypeScript SDK, build the link with the SDK's citation helpers. See [Working with snippets](https://expand.ai/docs/highlights/snippets) and [TypeScript SDK](https://expand.ai/docs/typescript-sdk#citation-helpers). ### Can I show citation links to my own users? Yes, when the page is safe to share. Snapshot links are public, so anyone with a link can open it. Do not fetch private pages, secrets, or customer data you do not want shared. See [Who can read a Snapshot](https://expand.ai/docs/wayback-machine#who-can-read-a-snapshot). ## Agents ### What is the difference between hooks, the Expand Skill, and MCP? A hook routes an agent's built-in web-fetch tool through Expand. The Skill teaches the agent how to use Expand from its shell. MCP gives the agent Fetch as a native tool. See [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks). ### Which agents support hooks? OpenCode. Claude Code, Cursor, and Codex get the Skill and the MCP server instead. See [CLI: Commands](https://expand.ai/docs/cli-commands#expandai-init). ### Is installing the Skill enough? It can be, when the agent can read the Skill and run terminal commands. Add MCP when you want native tool calls, or when the agent should not depend on a shell. See [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks). ### Should an agent use `expandai login` or `EXPAND_API_KEY`? For an agent on your machine, run `expandai login` once. The CLI and the local MCP server read that sign-in. Use `EXPAND_API_KEY` for the TypeScript SDK, the API, servers, and CI. See [Connect your agent (MCP)](https://expand.ai/docs/mcp-server#sign-in) and [Authentication](https://expand.ai/docs/authentication). ### What is `/start.md`? `/start.md` is the setup guide for agents, as one Markdown document. Give it to an agent and the agent picks the right setup for itself. See [start.md](https://expand.ai/docs/docs-for-agents#startmd). ### Why can't my agent see the Skill or the MCP server? Most often the install ran in a different directory, the agent needs a restart, or the MCP configuration landed in a different scope than the agent is using. See [Troubleshooting](https://expand.ai/docs/mcp-server#troubleshooting). ## Billing and tiers ### Is Expand billed by tokens? No. Expand bills in prepaid credits per delivered output, not per token. See [Pricing & Usage](https://expand.ai/docs/pricing-and-usage). ### What does a Fetch cost? `1` credit, and `1,000` credits cost `$1`. Highlights adds `1` credit, each converted document page adds `1`, each vision-model page adds `5`, a video adds `5`, and transcribed audio adds `10` a minute. See [Credit costs](https://expand.ai/docs/pricing-and-usage#credit-costs). ### Am I charged when a Fetch does not return a page? No. You pay for delivered content, not for attempts. The one surprise is that an origin's `404` or `500` page is a real capture, so it costs `1` credit like any other page. Check `originError` before you treat the content as the page you asked for. See [What you are charged for, by outcome](https://expand.ai/docs/pricing-and-usage#what-you-are-charged-for-by-outcome). ### How do I tell an Expand failure from a dead URL? Check the status and the error body together. A browser page Fetch can return `200` with an `originError` when the origin served an error page, or a `FetchOriginError` when it could not be reached. A file Fetch can instead return `404`, `410`, or `502` with `FetchOriginHttpError` when the origin refuses the file. Other non-2xx responses describe API or capture errors. See [When the origin fails](https://expand.ai/docs/errors-and-retries#when-the-origin-fails). ### What are credits? Credits are Expand's prepaid unit. You buy them at `$1` per `1,000`, and delivered work draws them down. A Fetch costs `1` credit. Your first organization starts with `10,000` welcome credits. See [Credits and balance](https://expand.ai/docs/pricing-and-usage#credits-and-balance). ### Do my credits expire? Purchased credits stay on your balance until you spend them. Welcome credits expire 365 days after they are granted. See [Credits and balance](https://expand.ai/docs/pricing-and-usage#credits-and-balance). ### What happens when I run out of credits? Expand checks your balance before it admits a request, so a request with no credits behind it is refused rather than run and billed. Buy credits in Billing and the block clears immediately. See [Access when credits run out](https://expand.ai/docs/pricing-and-usage#access-when-credits-run-out). ### Are tiers pricing plans? No. Tiers raise the browser concurrency included with your account as you spend credits and time passes. They do not change what anything costs. See [Concurrency tiers](https://expand.ai/docs/tiers). ### Why am I not in the next tier after spending enough? Tiers 2 to 5 need both a spend threshold and a number of days since your first transaction. If the spend is there and the days are not, the next tier is not active yet. See [Why am I not in the next tier?](https://expand.ai/docs/tiers#why-am-i-not-in-the-next-tier). ### What is browser concurrency? The number of browser captures your organization can run at once. At the limit, a single Fetch is refused with `429 FetchConcurrencyLimitExceeded` and Batched Fetch queues instead. See [Limits](https://expand.ai/docs/limits#concurrency-and-rate). ### Do provisioned browsers raise my tier? No. Provisioned browsers add to your total browser concurrency and leave your tier as it is. See [Provisioned browsers](https://expand.ai/docs/tiers#provisioned-browsers). ### When does a change to provisioned browsers take effect? Adding browsers takes effect immediately. Removing browsers takes effect at the end of the month you paid for. See [Provisioned browser concurrency](https://expand.ai/docs/pricing-and-usage#provisioned-browser-concurrency). ## Reference and machine-readable docs ### Why is the API reference separate from the docs? The docs explain what each product does and how to use it. The API reference holds the exact request and response schemas, errors, and generated examples. Open the [API reference](https://expand.ai/api-reference). ### Where are the exact request and response schemas? In the [API reference](https://expand.ai/api-reference). The docs link to it instead of repeating every schema. ### Can an agent read the docs as Markdown? Yes. Every docs and API reference page is available as Markdown, and `start.md`, `llms.txt`, and `llms-full.txt` give an agent a compact way in. See [Any page as Markdown](https://expand.ai/docs/docs-for-agents#any-page-as-markdown). ### What is the difference between `/start.md`, `llms.txt`, and `llms-full.txt`? `/start.md` is the setup guide for agents. `llms.txt` is a compact index of the docs. `llms-full.txt` is the whole docs bundle, for agents that need more context. See [start.md](https://expand.ai/docs/docs-for-agents#startmd), [llms.txt](https://expand.ai/docs/docs-for-agents#llmstxt), and [llms-full.txt](https://expand.ai/docs/docs-for-agents#llms-fulltxt). ### Can API reference pages be read as Markdown? Yes. Append `.md` to the page's path, or request the page with `Accept: text/markdown`. See [Any page as Markdown](https://expand.ai/docs/docs-for-agents#any-page-as-markdown). ## Next steps - [Pricing & Usage](https://expand.ai/docs/pricing-and-usage): credits and what each operation costs. - [Concurrency tiers](https://expand.ai/docs/tiers): how tiers are reached, with worked examples. - [Limits](https://expand.ai/docs/limits): concurrency, time, and size limits. - [Docs for agents](https://expand.ai/docs/docs-for-agents): read the docs as Markdown. --- # Get started with Expand (https://expand.ai/docs) Send a page URL and get Markdown with tables and icons intact, plus the page's hidden JSON as State JSON. Each Markdown line links to its source. Set up your agent, or run your first Fetch from the CLI, the TypeScript SDK, or the API. Expand Fetch is an API for agents that read the web. You send a URL. Expand opens it in a real browser, and our model Iris looks at the rendered page instead of guessing from the HTML. You get the page back as Markdown with tables, icons, and layout intact, plus the page's hidden JSON as [State JSON](https://expand.ai/docs/fetch#state-json). Every line links back to the spot on the page it came from, so you always know what your agent read. [Why Expand](https://expand.ai/docs/why-expand) explains what ordinary Markdown tools drop, and how Expand keeps it. Paste this into your coding agent to set up Expand: ```txt Read https://expand.ai/start.md and set up Expand. Fetch it as raw text with curl. A summarizing web-fetch tool drops the exact commands. ``` To configure one agent yourself, [set up the hosted MCP server](https://expand.ai/docs/mcp-server). Prefer code? The [Quickstart](https://expand.ai/docs/quickstart) runs your first Fetch from the CLI, the TypeScript SDK, or the API. ## Building blocks Fetch is one API, and everything else runs on it. Iris and X-Ray work on browser pages. The Wayback Machine keeps a replayable [Snapshot](https://expand.ai/docs/fetch#snapshot--evidence) of each browser page you fetch, and [Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight) finds the snippets in a page that answer a question. Crawl runs Fetch over a whole site, and WebSync keeps running it to catch changes. - [Fetch](https://expand.ai/docs/fetch) turns a URL into Markdown, with State JSON and a replayable Snapshot for browser pages. - [Iris](https://expand.ai/docs/fetch/iris) writes the Markdown from the rendered page. - [X-Ray](https://expand.ai/docs/fetch/x-ray) finds the data behind the page. - [Wayback Machine](https://expand.ai/docs/wayback-machine) saves Snapshots to replay and cite. - [Highlights](https://expand.ai/docs/highlights) finds cited passages that answer a query. - [Crawl](https://expand.ai/docs/crawl) runs Fetch across a site. - [WebSync](https://expand.ai/docs/websync) keeps a crawl up to date. --- # Limits (https://expand.ai/docs/limits) How many captures you can run at once, how long a request can take, and how large a page, file, batch, or crawl can be. Every limit in one place. When you hit one, the error names it and usually says when to try again. What to do next is on [Errors & retries](https://expand.ai/docs/errors-and-retries). ## Concurrency and rate **Browser concurrency** is how many browser captures your organization can run at the same time. Your tier includes 2 browsers at Tier 0 and up to 100 at Tier 5. Tiers rise on their own as you spend credits and time passes, and you can add provisioned browsers on top. The table is on [Concurrency tiers](https://expand.ai/docs/tiers). **Note:** Converting a PDF or other file does not use a browser, so document conversions do not count against your browser concurrency. When you are at your limit, a single Fetch is refused before it starts with `429 FetchConcurrencyLimitExceeded`. The response carries `maxConcurrency`, your own limit, and a `Retry-After` of 20 seconds. [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch) queues under the same limit instead of refusing. **Recommendation:** Send many URLs as one Batched Fetch rather than many single Fetches at once. Cache responses instead of fetching the same URL twice. **Throttling** is separate from concurrency. A throttled request gets `429 TooManyRequests` with a `Retry-After` header, and the same request succeeds once the wait is over. The TypeScript SDK waits and retries for you. Without an SDK, back off as described on [Errors & retries](https://expand.ai/docs/errors-and-retries#without-an-sdk). If these limits are too low for you, email [support@expand.ai](mailto:support@expand.ai). ## Time | What | Limit | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | A Fetch of a web page | One 150-second budget per request, from acceptance to delivery. A `504 FetchCaptureTimeout` or `529 FetchCapacityTimeout` arrives by 155 seconds. | | A PDF, other file, or video | Up to 55 minutes once a worker starts. The wait for a worker uses the 150-second budget. | | Each item in a Batched Fetch | Its own 150-second budget once it is scheduled. | | The TypeScript SDK | `timeoutMs` defaults to `60000` for the whole call, retries included. | **Recommendation:** Set your HTTP client timeout to 180 seconds for a page, and long enough for a document to finish. In the TypeScript SDK, raise `timeoutMs`. See [Choosing `timeoutMs`](https://expand.ai/docs/typescript-sdk#choosing-timeoutms). How the budget works, and what a `504` or `529` means, is on [Readiness and timing](https://expand.ai/docs/fetch/browser-rendered-pages#readiness-and-timing). ## Size | What | Limit | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A web page | A page too large to capture returns `413 FetchPageTooLarge` with `limitKind` (`transfer_bytes`, `domsnapshot_bytes`, or `document_frame_bytes`), the limit, and what was measured. It is not charged. | | A PDF or other file | No default page limit. Set `documentConfig.maxPages` to cap how many pages a Fetch converts. A file over the limit returns `413 FetchPageTooLarge` with `limitKind` set to `document_pages`. | | Audio and video | `documentConfig.maxAudioSeconds` defaults to `3600` and accepts `1` to `14400`. Four hours is the ceiling. A YouTube video with captions ignores it. Over the limit returns `413 FetchPageTooLarge` with `limitKind` set to `audio_seconds`. | | A Snapshot preview | `maxBytes` accepts `0` to `65536`. A preview of an archive over the server's read budget returns `413`. The full content link still works. | ## Batches, Highlights, and crawls | What | Limit | | -------------------------- | -------------------------------------------------------------------------------------------------------------------- | | Batched Fetch results page | `limit` defaults to `10` and accepts `1` to `100`. | | Idempotency keys | Up to 255 characters. | | Highlights | `maxResults` defaults to `5` and accepts `1` to `50`. `minScore` defaults to `0.6` and accepts `0` to `1`. | | Crawl sources | `1` to `1,000` per request. Up to `32` may set their own `maxPages`. | | Crawl size | `maxPages` defaults to `25,000` and accepts up to `1,000,000`. `maxDepth` defaults to `50` and accepts `0` to `100`. | | Crawl event content | 64 KiB per event. The full capture is always readable by its `captureId`. | | WebSync renders | `sync.budget.rendersPerDay` caps the pages a synced crawl renders in a day. You set it. | | Provisioned browsers | Up to `1,000` per organization. | ## Credits Your first organization starts with `10,000` welcome credits, which expire 365 days after they are granted. Purchased credits do not expire. A request made with no credits left is refused before any work starts and is never charged. See [Pricing & Usage](https://expand.ai/docs/pricing-and-usage). ## Next steps - [Errors & retries](https://expand.ai/docs/errors-and-retries): what to do when you hit a limit. - [Concurrency tiers](https://expand.ai/docs/tiers): how tiers and provisioned browsers work. - [Pricing & Usage](https://expand.ai/docs/pricing-and-usage): what each operation costs. --- # Connect your agent (MCP) (https://expand.ai/docs/mcp-server) Add Expand's hosted MCP server to Claude Code, Codex, Cursor, or any MCP client. Nothing to install, and you sign in through the browser. Expand runs a hosted MCP server. You point your agent at one URL and approve it in the browser. Your agent can then fetch any page as rendered Markdown, with a citation link for every line. ```txt https://api.expand.ai/v1/mcp ``` There is nothing to install. No Node, no CLI, and no API key in a config file. The hosted server is the same server the CLI runs locally with `expandai mcp run`, and it calls the same Fetch API, so your agent gets a real browser, JavaScript execution, [State JSON](https://expand.ai/docs/fetch#state-json), and a link back to the page for every line. **Recommendation:** The fastest route is the [one-prompt setup](https://expand.ai/docs/quickstart#run-fetch-in-seconds), which also installs the Expand Skill and, for OpenCode, a web-fetch hook. Use this page when you want to add the MCP server yourself. ## Add it to your agent **Claude Code** ```bash claude mcp add --transport http expand https://api.expand.ai/v1/mcp ``` Then run `/mcp` inside a Claude Code session and pick **expand** to sign in. Your browser opens, you approve, and the tools are live. **Codex** ```bash codex mcp add expand --url https://api.expand.ai/v1/mcp codex mcp login expand ``` **Cursor** Add this to `.cursor/mcp.json` in your project, or to `~/.cursor/mcp.json` for every project: ```json { "mcpServers": { "expand": { "url": "https://api.expand.ai/v1/mcp" } } } ``` Sign in when Cursor asks. **Other clients** Any client that takes a Streamable HTTP MCP server takes the URL directly. Put it in the client's MCP config and sign in when the client asks. For a client that cannot sign in through the browser, see [Use an API key instead](https://expand.ai/docs/mcp-server#use-an-api-key-instead). ## Check it worked Ask your agent to fetch something that a plain web-fetch tool cannot read: ```txt Use Expand to fetch https://news.ycombinator.com and show me the first 10 lines of Markdown. ``` A working setup returns Markdown and a **Sources** block that links to Expand's Snapshot of the page. If the agent returns an empty page or raw HTML, it used its own fetch tool rather than Expand's. ## Sign in The first tool call starts the sign-in. Your agent opens a browser, you approve, and the client stores the token. There is no API key to copy and nothing secret written into your agent's config file. If a client loses its authorization, run its sign-in step again (`/mcp` in Claude Code, `codex mcp login expand` in Codex) and the browser flow repeats. ### Use an API key instead For CI, scripts, or a client that cannot sign in through the browser, send an Expand API key as a bearer token in the `Authorization` header of the client config. The config entry, and where to create a key, are on [Authentication](https://expand.ai/docs/authentication#use-the-key). ## Tools your agent gets | Tool | Definition | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `fetch` | Fetches a URL in a real browser and returns the [Main Markdown](https://expand.ai/docs/fetch#main-markdown), the State JSON, and the Snapshot, with a ready-made citation block. Page meta tags only with `includeMeta`. | | `get_snapshot` | Read a public Snapshot catalog, or add `highlights` for [Snapshot Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight#snapshot-highlights). Highlights a page you already fetched, by Snapshot ID, without a new Fetch. | | `fetch_appendix` | Reads the navigation, footers, and related-content blocks that were moved to the [Appendix](https://expand.ai/docs/fetch#appendix), from the stored Snapshot. | Full parameters and return shapes are on [MCP tools & resources](https://expand.ai/docs/mcp-tools-and-resources). ## Troubleshooting **The client cannot connect, or reports a closed connection.** Confirm the URL is exactly `https://api.expand.ai/v1/mcp` and that the client treats it as an HTTP server, not a command to run. A client set up to start a local process reports a closed connection when there is no such process. **The agent still uses its own web fetch.** Some clients prefer their built-in tool. Name Expand in the prompt. In OpenCode, install the web-fetch hook so the built-in tool goes through Expand. See [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks). ## When you want more than MCP The hosted server covers the common case. The CLI adds what a remote server cannot do: routing your agent's own web-fetch tool through Expand, fetching from a terminal, and running through a proxy set in your environment. See [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks). ## Next steps - [MCP tools & resources](https://expand.ai/docs/mcp-tools-and-resources): full parameters and return shapes. - [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks): route your agent's own web-fetch tool through Expand, or fetch from a terminal. - [Authentication](https://expand.ai/docs/authentication): use an API key instead of browser sign-in. - [Docs for agents](https://expand.ai/docs/docs-for-agents): give your agent Expand's docs, through start.md, llms.txt, or Markdown pages. --- # MCP tools & resources (https://expand.ai/docs/mcp-tools-and-resources) The three tools the Expand MCP server gives an agent, with every parameter, default, and result field, plus the two documentation resources. The Expand MCP server gives an agent three tools. `fetch` is a Fetch in JSON Mode. `get_snapshot` reads a public Snapshot catalog, or returns [Snapshot Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight#snapshot-highlights) when given a `highlights` object. `fetch_appendix` reads the [Appendix](https://expand.ai/docs/fetch#appendix) of a page the agent already fetched. Fetch and Highlights results carry a ready-made citation block. Two documentation resources point the agent at the setup guide. The server runs in two places, with the same tools either way. ## Hosted ```txt https://api.expand.ai/v1/mcp ``` Streamable HTTP, nothing installed. The agent signs in through the browser with OAuth, or sends an Expand API key as `Authorization: Bearer xpnd_...`. The per-client setup is on [Connect your agent (MCP)](https://expand.ai/docs/mcp-server). ## Local over stdio ```bash expandai mcp run ``` Agents launch this command from their MCP config, not from a terminal. It uses the credentials from `expandai login`, so no key is written into the config. `expandai init` or `expandai mcp install ` writes the entry for you. For Claude Code and Cursor it looks like this: ```json { "mcpServers": { "expandai": { "command": "expandai", "args": ["mcp", "run"] } } } ``` **Note:** When `expandai` is not on `PATH` at install time, the entry runs `npx -y expandai mcp run` instead. Where each agent's config lives is on [CLI: Commands](https://expand.ai/docs/cli-commands#expandai-mcp-install-host-scope). ## Tools All three tools are read-only and non-destructive. | Tool | Definition | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `fetch` | Fetch a URL in a real browser and return a JSON object with the [Main Markdown](https://expand.ai/docs/fetch#main-markdown), the [State JSON](https://expand.ai/docs/fetch#state-json), the [Snapshot](https://expand.ai/docs/fetch#snapshot--evidence), and, with `highlights`, [Fetch Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight#fetch-highlights). | | `get_snapshot` | Read a public Snapshot catalog by `snapshotId`, or add `highlights` for relevant passages. No new Fetch. | | `fetch_appendix` | The Appendix of a page the agent already fetched, whole or one block by evidence ID. No new Fetch. | ### `fetch` Calls `POST /v1/fetch/json` and returns the JSON object. The agent's tool description tells it to use `fetch` before `curl` or a built-in web-fetch tool on any public page, and to skip it for localhost, private hosts, and JSON APIs. | Option | Definition | Default | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | `url` | The page to fetch. | Required | | `highlights` | A `highlights` object. When set, the result also carries Fetch Highlights under `data.highlights.snippets`. | None | | `includeMeta` | Include the page's meta tags: title, description, Open Graph. | `false` | | `includeAppendix` | Include the [Appendix](https://expand.ai/docs/fetch#appendix). | `false` | | `includeJson` | Include the [State JSON](https://expand.ai/docs/fetch#state-json). | `true` | | `appendixMarkers` | How verbose the `[appendix omitted: …] {N}` markers in the Markdown are. `compact`, `detailed`, which adds a landmark label and a preview, or `off`. | `compact` | The `highlights` object: | Option | Definition | Default | | ------------ | --------------------------------------------------- | -------- | | `query` | The query. | Required | | `maxResults` | Most snippets to return, `1` to `50`. | `5` | | `minScore` | Lowest relevance score a snippet needs, `0` to `1`. | `0` | **Note:** `minScore` defaults to `0` here and to `0.6` in the CLI, the TypeScript SDK, and the API. An agent sees every snippet up to `maxResults` unless it raises the score. How to tune both is on [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight#tuning-the-results). ```json { "name": "fetch", "arguments": { "url": "https://docs.stripe.com/webhooks", "includeMeta": true, "highlights": { "query": "signature verification", "maxResults": 5 } } } ``` | Field | Definition | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `meta` | The Snapshot. `meta.snapshotId` is the ID to pass to `get_snapshot` and `fetch_appendix`. `meta.playground` is the capture link. | | `markdown` | The [Main Markdown](https://expand.ai/docs/fetch#main-markdown), with the `citationBlock` appended after a `---` rule. | | `json` | The State JSON, one entry per piece of evidence. | | `data.highlights.snippets` | Fetch Highlights, when the call had a `highlights`. See [Snippets and citations](https://expand.ai/docs/mcp-tools-and-resources#snippets-and-citations). | | `citationBlock` | A `**Sources**` block with up to three capture links, ready to paste at the end of an answer. | | `citationInstructions` | How to use the block. The agent is told to follow it exactly. | When the origin never answered, the tool returns an error that names the failure and whether a retry makes sense, instead of content. See [Errors & retries](https://expand.ai/docs/errors-and-retries). ### `get_snapshot` Without `highlights`, returns the public catalog with `collections`, `artifacts`, and `links`. No authentication is needed. With `highlights`, calls `POST /v1/snapshots//highlights` and returns [Snapshot Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight#snapshot-highlights) for a page the agent already fetched. The page is not fetched again. Highlights requires authentication and costs one Highlights credit. | Option | Definition | Default | | ----------------- | --------------------------------------------------------------------------- | -------- | | `snapshotId` | `meta.snapshotId` from an earlier `fetch` result. | Required | | `highlights` | The same `highlights` object as `fetch`: `query`, `maxResults`, `minScore`. | None | | `includeMarkdown` | Highlight the Main Markdown. | `true` | | `includeJson` | Highlight the State JSON. | `true` | | `includeAppendix` | Highlight the Appendix. | `false` | The layer options apply when `highlights` is present. Turning all three layers off returns `400 BadRequest`. **Warning:** Pass `meta.snapshotId`, the capture's own ID. `meta.resources.snapshotId` is the public Snapshot ID and returns `404 FetchSnapshotHighlightsNotFound`. See [Two IDs](https://expand.ai/docs/wayback-machine#two-ids). | Field | Definition | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `snapshotId` | The ID that was searched. | | `highlights.snippets` | The snippets. See [Snippets and citations](https://expand.ai/docs/mcp-tools-and-resources#snippets-and-citations). | | `response` | The original page's URL and response details. | | `durationMs` | How long Highlights took. | | `playground` | The capture link for the whole Snapshot. | | `citationBlock`, `citationInstructions` | As on `fetch`. | ### `fetch_appendix` Reads the Appendix of a page the agent already fetched. Where Fetch moved navigation, a footer, or related links out of the Main Markdown, the Markdown carries a marker: ```txt [appendix omitted: 14 links · ~320 tokens] {612} ``` | Option | Definition | Default | | ------------ | -------------------------------------------------------------------------------------------- | ------------------ | | `snapshotId` | `meta.snapshotId` from an earlier `fetch` result, or the capture URL from `meta.playground`. | Required | | `id` | The evidence ID from a marker, such as `612`. Returns that block only. | The whole Appendix | | Field | Definition | | --------------------------------------- | --------------------------------------------------------------------------- | | `markdown` | The requested Appendix content as Markdown. | | `evidenceId` | The `id` that was asked for, when there was one. | | `playground` | The capture link for the whole Snapshot. | | `citationBlock`, `citationInstructions` | As on `fetch`. The link carries `?id=` when one block was read. | The tool reads the public Snapshot routes on the [Playground](https://expand.ai/docs/wayback-machine/playground-and-replay) host. It needs no API key and starts no new Fetch. **Note:** The counts in a marker say how much is behind it. The tool description tells the agent to read a block only when the task needs it, because the Appendix can be larger than the page. ### Snippets and citations Every snippet from `fetch` or `get_snapshot` has these fields: | Field | Definition | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `source` | Where the snippet came from: `markdown`, `appendix`, or `statejson`. | | `text` | The matched passage. | | `json` | The matched value, when `source` is `statejson`. | | `score` | Relevance, `0` to `1`. | | `location` | Where on the Snapshot the passage is, including `evidenceId`. | | `citationUrl` | A [citation link](https://expand.ai/docs/wayback-machine/playground-and-replay#citation-links) that opens the exact passage on the capture. | The server builds the links so the agent does not have to. `citationBlock` lists up to three of them as a `**Sources**` block, and on `fetch` the same block is appended to `markdown`, so a summary of the page carries its sources with it. State JSON snippets are left out of the block, because extracted page state is not a quotable passage. What a snippet contains and how to read beyond it is on [Working with snippets](https://expand.ai/docs/highlights/snippets). ## Documentation resources The server can also serve two resources. | Resource | Name | Contents | | ---------------- | ------------------ | ----------------------------------------------------------------------------------------------------- | | `expand://about` | About expand.ai | Links to the website, the docs, and the dashboard. | | `expand://start` | Expand Start Guide | The link to `https://expand.ai/start.md` and the prompt that runs it. The guide itself is not copied. | The hosted server always serves them. The local server serves them only with `--docs`: ```bash expandai mcp run --docs ``` **Note:** `expandai init` and `expandai mcp install` register `expandai mcp run` without `--docs`, so do not assume `expand://start` exists on a local server. The guide itself is at [`/start.md`](https://expand.ai/start.md). See [Docs for agents](https://expand.ai/docs/docs-for-agents). ## Authentication The local server uses the same credentials as the CLI, stored by `expandai login`. The hosted server signs the agent in through the browser, or takes an API key as a bearer token, which it verifies against your organization's keys. Details are on [Authentication](https://expand.ai/docs/authentication). ## Next steps - [Connect your agent (MCP)](https://expand.ai/docs/mcp-server): add the hosted server to your agent. - [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks): what the local CLI adds, and what the Skill teaches. - [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight): Inline and Snapshot Highlights, and how to tune them. - [API reference](https://expand.ai/api-reference/fetch/fetch.fetchJson): the exact request and response schemas behind `fetch`. --- # Pricing & Usage (https://expand.ai/docs/pricing-and-usage) What each operation costs in credits, how credits are bought and spent, what is never charged, and how browser concurrency is priced. Expand runs on prepaid credits. You buy credits up front, and delivered work draws them down. `1,000` credits cost `$1`, so one credit is `$0.001`. One successful Fetch costs `1` credit. **[Credits and balance](https://expand.ai/docs/pricing-and-usage#credits-and-balance)** Start with `10,000` welcome credits and buy more in packs of `1,000`. **[Credit costs](https://expand.ai/docs/pricing-and-usage#credit-costs)** A Fetch is `1` credit. Highlights, document pages, and transcripts add credits on top. **[Browser concurrency](https://expand.ai/docs/pricing-and-usage#browser-concurrency)** Your tier sets how many browsers you can run at once. Provisioned browsers add more, at `$1` per browser per month. **[What you are charged for](https://expand.ai/docs/pricing-and-usage#what-you-are-charged-for-by-outcome)** Failed, blocked, and empty work costs `0` credits. You pay for content you received. Your balance and the `Buy credits` button are in your dashboard under Settings, Billing. Request activity over time is in your dashboard under Usage. How tiers are reached is on [Concurrency tiers](https://expand.ai/docs/tiers). ## Credit costs Every priced operation is quoted in credits. | Operation | Credits | Definition | | ------------------------------ | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Fetch | `1` | One delivered page, file, or video. The base unit. Every other row is added on top of it. | | Highlights | `1` | One Highlights request, inline with a Fetch or on a Snapshot you already have. See [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight). | | Converted document page | `1` per page | Every page a [PDF or other file](https://expand.ai/docs/fetch/documents) converts. An image counts as one page. | | Vision-model page | `5` per page | A page read by a vision model instead of a text layer, such as a scanned PDF or any image. On top of the converted page. | | Video | `5` per video | A [YouTube video](https://expand.ai/docs/fetch/youtube) that delivered a transcript, whatever its length. | | Transcribed audio | `10` per minute | Audio files, and videos with no caption track. Rounded up, one-minute minimum. | | Crawl page render | `1` per page | A page [Crawl](https://expand.ai/docs/crawl) renders in a browser. A freshness check that does not render is free. | | Failed, blocked, or empty work | `0` | Nothing was delivered, so nothing is charged. | **Note:** Browser concurrency is not priced in credits. Your tier includes a number of browsers, and provisioned browsers cost `$1` per browser per month on your card. See [Browser concurrency](https://expand.ai/docs/pricing-and-usage#browser-concurrency). Expand bills per delivered output, not per token. For a committed monthly allowance, bulk rates, or enterprise terms, email [support@expand.ai](mailto:support@expand.ai). ## Credits and balance Your organization holds one prepaid credit balance. Delivered work draws it down, and buying credits tops it up. Your first organization starts with `10,000` welcome credits, worth `$10`. Welcome credits expire 365 days after they are granted. Purchased credits stay on your balance until you spend them. Billing shows your balance as two numbers: - **Available credits.** Credits you can spend now. - **Expiring soon.** Credits set to expire within 30 days. Buy more with `Buy credits`. Choose a number of credits, any multiple of `1,000` up to `10,000,000`, and pay by card in a hosted checkout. The credits land on your balance as soon as the payment is confirmed. **Note:** Anonymous use of the landing page and the Playground runs on separate demo credits. It does not touch your balance. ## Usage metering A Fetch costs `1` credit. Highlights on the same request add `1` more. If the URL serves a file rather than a page, the pages it converts are added on top. Each requested output is charged on its own: | What you fetch | Credits | | --------------------------------------------------- | ------------------------------------------------------ | | A page | `1` | | A page with Highlights | `2` | | An office file, ebook, CSV, JSON, XML, or text file | `1`. These convert without a page count. | | A PDF with a text layer | `1`, plus `1` per converted page | | A scanned PDF, or any image | `1`, plus `1` per page, plus `5` per vision-model page | | An audio file | `1`, plus `10` per transcribed minute | | A YouTube video with captions | `6`, the Fetch plus the video | | A YouTube video with no captions | `6`, plus `10` per transcribed minute | Document conversion is charged only on a delivered result. A Fetch refused for exceeding a limit, for an unconvertible format, or because the origin returned no file is charged nothing. **Recommendation:** Set `documentConfig.maxPages` to cap the pages a Fetch may convert, and `documentConfig.ocr` to `"never"` to skip the vision model. There is no default page limit. See [Limiting what gets converted](https://expand.ai/docs/fetch/documents#limiting-what-gets-converted). The JSON object reports what each Fetch was charged for in `data.usage`: `documentPages`, `ocrPages`, `audioSeconds`, and `videos`. The Usage view in your dashboard shows request activity over time. ### What you are charged for, by outcome You pay for delivered content, not for attempts. Every Fetch holds credits up front and releases the hold when the request settles, so a request that produced nothing costs nothing. | Outcome | Charged | Why | | ----------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | The page was captured | Yes | Content was delivered. | | The origin answered with an HTTP error status (`400` and above) and the page was captured | Yes | Error pages are still content, such as a soft 404, a maintenance notice, or a status page. You receive the page plus `originError` with the status. | | The origin could not be reached | No | No page was delivered. The response is `200` with a `FetchOriginError` in place of content. | | The origin refused the file, or it was missing or removed | No | No file was delivered. The error is `FetchOriginHttpError` at `404` or `410` for a missing or removed file, and `502` otherwise. | | The page blocked the capture (`403 FetchBlocked`) | No | Expand did not deliver the page. | | The page exceeded a size limit (`413 FetchPageTooLarge`) | No | Expand did not deliver the page. | | Expand failed (`500`, `503`) | No | Our problem, not your charge. | | A crawl freshness check that did not render | No | An internal check, never a delivered page. | **Warning:** An origin's `404` or `500` page is a real capture, so it costs `1` credit like any other page. Check `originError` before you treat the content as the page you asked for. Highlights are charged on the same basis. Delivered snippets cost `1` credit, and a request that never produced a capture costs nothing. Which errors are charged, and which to retry, is on [Errors & retries](https://expand.ai/docs/errors-and-retries#what-you-are-charged-for). ## Browser concurrency Browser concurrency is how many browser captures your organization can run at once. It has two parts: ```txt Total browser concurrency = tier concurrency + provisioned browsers ``` ### Tier overview Your tier sets the browsers included with your account, from 2 at Tier 0 to 100 at Tier 5. Tiers rise on their own as you spend credits and time passes. The table, and how tiers are reached, is on [Concurrency tiers](https://expand.ai/docs/tiers). ### Provisioned browser concurrency If you need more browsers before your next tier, add provisioned browsers in Billing. Each one costs `$1` per month, billed to the card on file. The price is a fixed subscription, not a credit cost, and the browsers are added on top of your tier. You can provision up to `1,000`. | Change | When it takes effect | What you pay | | ------------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------- | | Adding browsers | Immediately | A prorated charge for the rest of the current month right away, then `$1` per browser per month | | Removing browsers | At the end of the month you paid for | Nothing more. The browsers you paid for stay until the month ends, and nothing is refunded | | Setting the count to zero | At the end of the month you paid for | Nothing more | **Note:** If your organization has no card on file, Billing sends you to a hosted checkout first. The browsers appear once the payment is confirmed. At your limit, a single Fetch is refused with `429 FetchConcurrencyLimitExceeded` and [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch) queues instead. See [Limits](https://expand.ai/docs/limits#concurrency-and-rate). Enterprise organizations get their browser concurrency from their contract and do not buy browsers here. ## Access when credits run out Expand checks your balance before it admits a request. With credits on your balance, the request runs. With none, the request is refused before any work starts, so you are never charged for a request Expand declined to run. Buying credits clears the block immediately. ## Next steps - [Concurrency tiers](https://expand.ai/docs/tiers): the tier table, how tiers are reached, and worked examples. - [Limits](https://expand.ai/docs/limits): concurrency, time, and size limits in one place. - [Errors & retries](https://expand.ai/docs/errors-and-retries): which errors are charged, and which to retry. - [FAQ](https://expand.ai/docs/faq): short answers to billing questions. --- # Python SDK (https://expand.ai/docs/python-sdk) The official Expand Python SDK is coming soon. Until it is ready, use the [HTTP API](https://expand.ai/api-reference) directly from Python. For supported integrations today, use the [TypeScript SDK](https://expand.ai/docs/typescript-sdk), [CLI](https://expand.ai/docs/cli-commands), or [MCP tools](https://expand.ai/docs/mcp-tools-and-resources). --- # Quickstart (https://expand.ai/docs/quickstart) Run your first Fetch from the CLI or a coding agent in one step, then pick the surface you will build on. A Fetch takes a URL and returns the rendered page. In Markdown Mode you get Markdown text. In JSON Mode you get a JSON object with the [Main Markdown](https://expand.ai/docs/fetch#main-markdown), [State JSON](https://expand.ai/docs/fetch#state-json), page metadata, and the [Snapshot ID](https://expand.ai/docs/wayback-machine#two-ids) together. This page runs one Fetch, then shows the same call from a coding agent, the CLI, the TypeScript SDK, and the API. ## Run Fetch in seconds Run one command in your terminal, or paste the prompt into your coding agent. **CLI** ```bash npx expandai fetch https://news.ycombinator.com ``` **Agent** ```txt Read https://expand.ai/start.md and set up Expand. Fetch it as raw text with curl. A summarizing web-fetch tool drops the exact commands. ``` **Note:** You can try a Fetch from the CLI before you sign in. The CLI uses an anonymous preview session, which is rate limited. Run `expandai login` for normal use. Your first organization starts with `10,000` free credits, and a Fetch costs `1` credit (`1,000` credits are `$1`). See [Pricing & Usage](https://expand.ai/docs/pricing-and-usage). ## Ways to use Expand You can call Fetch from a coding agent, the CLI, the TypeScript SDK, the API, or the Playground. They all call the same API, so a browser page Fetch returns the same Main Markdown, State JSON, citation links, and Snapshot whichever way you call it. Pick the one that matches where you are. - Inside a coding agent, set the agent up once so it fetches pages through Expand on its own. - Testing by hand, run one CLI command. - Building a product, call the TypeScript SDK or the API and hand the result to your own users. - Checking what a Fetch saw, open the Playground. | Use it from | Best for | Setup | Auth | Docs | | -------------------------------------------------------------------------- | ----------------------------------- | ------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------- | | [Agents](https://expand.ai/docs/quickstart#agents) | Coding agents and MCP clients | Hosted MCP server, one URL | Browser sign-in or API key | [Connect your agent](https://expand.ai/docs/mcp-server) | | [CLI](https://expand.ai/docs/quickstart#cli) | Terminal fetches and scripts | `npm install --global expandai` | CLI login or `EXPAND_API_KEY` | [CLI commands](https://expand.ai/docs/cli-commands) | | [TypeScript SDK](https://expand.ai/docs/quickstart#typescript-sdk-and-api) | TypeScript apps | Package install | `EXPAND_API_KEY` | [TypeScript SDK](https://expand.ai/docs/typescript-sdk) | | [API](https://expand.ai/docs/quickstart#typescript-sdk-and-api) | Custom backends and other languages | HTTP client | `x-expand-api-key` | [API reference](https://expand.ai/api-reference) | | Playground | Visual inspection and Replay | Dashboard | Signed-in account | [Playground & Replay](https://expand.ai/docs/wayback-machine/playground-and-replay) | MCP is one way to connect an agent, not a separate product. Hooks, the Skill, and MCP work together. A hook routes the agent's own web-fetch tool through Expand, the Skill teaches the agent when to reach for Expand, and MCP gives the agent Fetch as tools it can call. Setup picks the right mix for each agent, so you do not have to assemble it yourself. To hand an agent Expand's docs instead of the live site, use the routes on [Docs for agents](https://expand.ai/docs/docs-for-agents). ## Agents Paste the setup prompt from [Run Fetch in seconds](https://expand.ai/docs/quickstart#run-fetch-in-seconds) into your coding agent. The agent reads the guide, asks which agents on your machine to configure, and runs `expandai init` for the ones you pick. Each one gets the Expand Skill and the MCP server, and OpenCode also gets a web-fetch hook. Claude Code, Codex, Cursor, and OpenCode are supported today. What each piece does, and the per-agent commands, are on [CLI: Skill & hooks](https://expand.ai/docs/cli-skill-and-hooks). To add only the hosted MCP server, with nothing installed, see [Connect your agent (MCP)](https://expand.ai/docs/mcp-server). **Note:** Agents read hook and MCP configuration when they start, so setup takes effect in a new session. After setup, confirm it with one prompt: ```txt Use Expand to fetch https://news.ycombinator.com. Show me the first 10 lines of Markdown and confirm the Fetch succeeded. ``` ## CLI ### Install Run it once with `npx expandai`, or install it globally: ```bash npm install -g expandai ``` ### Sign in Sign in once and the CLI keeps the session on your machine: ```bash expandai login ``` The browser opens so you can sign in or create an account. For CI, servers, and scripts, set `EXPAND_API_KEY` instead. Where to create a key, and how each surface sends it, is on [Authentication](https://expand.ai/docs/authentication). ### Fetch a page ```bash expandai fetch https://news.ycombinator.com ``` Markdown Mode is the default. The CLI prints the page as Markdown text. To get the JSON object instead, with the Main Markdown, State JSON, page metadata, and `meta.snapshotId` together, pass `--format json`: ```bash expandai fetch https://news.ycombinator.com --format json ``` **Note:** Use JSON Mode when you need the Snapshot ID, for replay or for [Snapshot Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight#snapshot-highlights). Every field in the response is on [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url). ## TypeScript SDK and API The TypeScript SDK and the API are the path into application code. Install the SDK with `npm install @expandai/sdk`. The API needs nothing installed. Both read your API key from `EXPAND_API_KEY`: ```bash export EXPAND_API_KEY="xpnd_..." ``` Create the key in the [dashboard](https://expand.ai/settings/api-keys) after signing in. How each surface sends it is on [Authentication](https://expand.ai/docs/authentication). Then run the first Fetch: **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://news.ycombinator.com' }) console.log(markdown) ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://news.ycombinator.com"}' ``` `POST /v1/fetch`, and `expand.fetch()` in the SDK, is Markdown Mode and returns Markdown text. `POST /v1/fetch/json`, and `expand.fetchJson()`, is JSON Mode and returns a JSON object with the Main Markdown, State JSON, page metadata, and `meta.snapshotId`. The exact schemas are in the [API reference](https://expand.ai/api-reference), and the full client is on [TypeScript SDK](https://expand.ai/docs/typescript-sdk). ## Troubleshooting - The CLI says you are not signed in. Run `expandai login`, then `expandai whoami` to confirm. - A request is refused for a missing key. Set `EXPAND_API_KEY`, or send the `x-expand-api-key` header on direct API calls. - `EXPAND_API_KEY` is set but the CLI uses another account. A stored sign-in takes precedence over the environment variable. Run `expandai logout` to use the key. - The page comes back empty or blocked. Try another URL, then see [Browser-rendered pages](https://expand.ai/docs/fetch/browser-rendered-pages). - You need the Snapshot ID. Use `--format json` on the CLI, or `POST /v1/fetch/json`. - You got an error code. [Errors & retries](https://expand.ai/docs/errors-and-retries) says what it means and whether to retry. ## Next steps - [Connect your agent (MCP)](https://expand.ai/docs/mcp-server): add the hosted MCP server by hand. - [CLI: Commands](https://expand.ai/docs/cli-commands): every CLI command and flag. - [TypeScript SDK](https://expand.ai/docs/typescript-sdk): the full client. - [Fetch: How Fetch works](https://expand.ai/docs/fetch#how-fetch-works): Main Markdown, State JSON, Appendix, and the Snapshot. --- # Concurrency tiers (https://expand.ai/docs/tiers) How tiers raise the browser concurrency included with your account, how they are reached, and what they do not change. Tiers set how many browsers your organization can run at once. Expand works out your tier from two numbers, the credits you have spent and the days since your first transaction. There is no plan to pick and no upgrade button to press. Your current tier is shown in your dashboard under Settings, Billing. How credits are priced is on [Pricing & Usage](https://expand.ai/docs/pricing-and-usage). If you need more browsers before the next tier, add [provisioned browsers](https://expand.ai/docs/tiers#provisioned-browsers). **[Automatic](https://expand.ai/docs/tiers#how-tiers-are-calculated)** Your tier is recomputed each time an operation settles. There is nothing to apply for. **[Spend and time](https://expand.ai/docs/tiers#how-tiers-are-calculated)** Tiers 2 to 5 need both a spend threshold and a number of days since your first transaction. **[Included browsers](https://expand.ai/docs/tiers#browser-concurrency)** Each tier raises the number of browsers included with your account. **[Provisioned browsers](https://expand.ai/docs/tiers#provisioned-browsers)** Need more browsers now? Add them on top of your tier for `$1` per browser per month. ## Tier table | Tier | Qualification | Included browser concurrency | | ---- | -------------------------------------------------- | ---------------------------: | | 0 | Default | 2 | | 1 | `$10` spent | 5 | | 2 | `$50` spent and 7 days since first transaction | 10 | | 3 | `$200` spent and 14 days since first transaction | 20 | | 4 | `$500` spent and 30 days since first transaction | 50 | | 5 | `$1,000` spent and 60 days since first transaction | 100 | Every new account starts at Tier 0. For Tiers 2 to 5, both conditions must be met. ## How tiers are calculated Expand calculates your tier from two values, the dollar value of the credits your organization has spent and the days since its first transaction. Your tier is the highest one whose spend and time conditions you both meet. - Tier 0 is the default until you qualify for a higher one. - Spend alone is not always enough. Tiers 2 to 5 also need days to pass. - Time alone is never enough. Every tier above 0 has a spend threshold. - The days count from your first settled operation, not from signup. ## What counts toward spend Spend is the value of the credits your organization has used, at the public rate of `$1` per `1,000` credits. Every settled operation adds its credits to your spend, and your first settled operation starts the day count. - Buying credits does not count. A purchase raises your balance, not your tier. The tier moves as you spend the credits. - Failed, blocked, or empty operations cost nothing and add nothing. - Spend is measured at the public rate whatever you paid per credit. - Organizations that started on our previous billing system keep the spend and the first-transaction date of the invoices they paid there. ## When your tier updates Your tier is recomputed each time an operation settles. When a settlement carries you across a threshold, the new browser concurrency applies to your next request. There is nothing to do. Settlement normally follows the operation within seconds. A Batched Fetch settles once the whole run finishes, so its spend lands when the run does. **Note:** If your spend is high enough but your tier has not changed, check the days. Most tier surprises come from the days-since-first-transaction condition. ## Browser concurrency The concurrency number in the tier table is the browser concurrency included with your tier. It is the number of browser captures your organization can run at once. ```txt Total browser concurrency = tier concurrency + provisioned browsers ``` | Term | Definition | | ------------------------- | --------------------------------------------- | | Tier concurrency | The browsers included with your current tier. | | Provisioned browsers | Paid browsers added on top. | | Total browser concurrency | The limit Fetch and Batched Fetch run under. | Browser concurrency is not a request rate limit. At your limit, a single Fetch is refused with `429 FetchConcurrencyLimitExceeded` and [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch) queues instead. See [Limits](https://expand.ai/docs/limits#concurrency-and-rate). ## Provisioned browsers If you need more browsers before the next tier, add provisioned browsers on top of your tier. They raise your total browser concurrency and leave your tier unchanged. Cost and timing are on [Provisioned browser concurrency](https://expand.ai/docs/pricing-and-usage#provisioned-browser-concurrency). ## Examples Each example follows the [tier table](https://expand.ai/docs/tiers#tier-table). ### Spend before time You spend `$50` on day 1. Tier 2 also needs 7 days since your first transaction, so until then you are at Tier 1. ### Time without spend Your account is 30 days old and has spent `$10`. Tier 4 needs `$500`, so you stay at Tier 1. ### Higher spend, waiting on time You have spent `$1,000` after 30 days. Tier 5 needs 60 days, so you are at Tier 4 until day 60. ### Provisioned browsers You are at Tier 2 with 10 included browsers and add 25 provisioned browsers. Your total browser concurrency is 35. ## Why am I not in the next tier? Work down this list: - Have you spent enough? See [What counts toward spend](https://expand.ai/docs/tiers#what-counts-toward-spend). - Have enough days passed since your first settled operation? - Have your operations settled? A Batched Fetch that is still running has not been counted yet. - Are you counting purchased credits? Only spent credits count. - Are you reading your tier concurrency, or your total after provisioned browsers? The most common cause is that the spend threshold is met and the time threshold is not. ## What tiers do not change - Tiers are not pricing plans. - Tiers do not change what an operation costs in credits. See [Pricing & Usage](https://expand.ai/docs/pricing-and-usage). - Tiers do not add or remove provisioned browsers. - Tiers do not set time or size limits. See [Limits](https://expand.ai/docs/limits). For browser concurrency beyond Tier 5, or a committed plan, email [support@expand.ai](mailto:support@expand.ai). ## Next steps - [Pricing & Usage](https://expand.ai/docs/pricing-and-usage): credits, costs, and provisioned browsers. - [Limits](https://expand.ai/docs/limits): what happens at your concurrency limit, and every other limit. - [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch): fetch many URLs within your concurrency. - [FAQ](https://expand.ai/docs/faq): short answers to billing questions. --- # TypeScript SDK (https://expand.ai/docs/typescript-sdk) Call Fetch, Highlights, Batched Fetch, Crawl, and WebSync from TypeScript with @expandai/sdk, and turn the results into citation links. `@expandai/sdk` is a typed client for the Expand API. It reads your API key from the environment, retries transient failures, and returns the same Markdown text and JSON object the API does. **npm** ```bash npm install @expandai/sdk ``` **pnpm** ```bash pnpm add @expandai/sdk ``` **bun** ```bash bun add @expandai/sdk ``` ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://news.ycombinator.com' }) ``` ## Install and auth Set `EXPAND_API_KEY` and the client picks it up. Pass `apiKey` only when the key comes from somewhere else, such as a secrets manager. The constructor throws when it finds no key, so a missing key fails at startup rather than on the first request. ```bash export EXPAND_API_KEY="xpnd_..." ``` ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient({ apiKey: process.env.EXPAND_API_KEY, timeoutMs: 180_000, }) ``` | Option | Definition | Default | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `apiKey` | Your API key, sent as the `x-expand-api-key` header. | `process.env.EXPAND_API_KEY` | | `baseUrl` | Where requests go. | `https://api.expand.ai` | | `timeoutMs` | The time budget for one call, across every attempt. See [Choosing `timeoutMs`](https://expand.ai/docs/typescript-sdk#choosing-timeoutms). | `60000` | | `maxRetries` | How many times a failed attempt is retried. | `2` | **Note:** `fetch()`, `fetchJson()` with Fetch parameters, `snapshots.get()`, and the Batched Fetch methods take `timeoutMs` and `maxRetries` in their final options argument to override the client's defaults for one call. `snapshots.get()` uses the same timeout and retry options. Other Crawl and Snapshot methods have their own options. To create a key, see [Authentication](https://expand.ai/docs/authentication). ## Method index | Method | Definition | Returns | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | | `fetch(params, options?)` | Markdown Mode. One URL in, Markdown text out. | `string` | | `fetchJson(params, options?)` | JSON Mode. [Main Markdown](https://expand.ai/docs/fetch#main-markdown), [State JSON](https://expand.ai/docs/fetch#state-json), the Snapshot, and Fetch Highlights. | `FetchJsonResponse` | | `snapshots.get(params, options?)` | Read the public catalog, or add `highlights` for relevant passages from the Snapshot. | `SnapshotCatalog` or `SnapshotHighlightsResponse`, inferred from `highlights` | | `batched(params, options?)` | Start a [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch). | The run's `id` | | `getBatched(id, options?)` | Poll a run for its status and results. | `GetBatchedResponse` | | `cancelBatched(id, options?)` | Stop a run. | `CancelBatchedResponse` | | `snapshots.get`, `snapshots.network`, `snapshots.preview` | Read a Snapshot's public catalog, its network requests, or a slice of one artifact. No API key needed. | See [Progressive disclosure](https://expand.ai/docs/typescript-sdk#progressive-disclosure) | | `crawls.create`, `crawls(id)`, `crawls.merge` | Start a crawl, pick one up by ID, or read several as one stream. | `CrawlResource` | | `captures(id)`, `pages(id)` | A capture's full content, or a page's version history. | Capture and page readers | | `close()` | Abort active requests and release the client's connections. | `Promise` | ## Fetch ### JSON Mode with `fetchJson` `fetchJson()` calls `/v1/fetch/json` and returns a JSON object: the Main Markdown in `markdown`, the State JSON in `json`, and the Snapshot in `meta`. Use it when your code reads the fields separately. When the origin never answered, the result is a `FetchOriginError` instead of content. Narrow with `isFetchOriginError()` before reading the fields. ```ts import { ExpandClient, isFetchOriginError } from '@expandai/sdk' const expand = new ExpandClient() const page = await expand.fetchJson({ url: 'https://www.figma.com/pricing/' }) if (isFetchOriginError(page)) { console.error(page.kind, page.retryable) } else { console.log(page.markdown) console.log(page.json) console.log(page.meta.playground) } ``` | Field | Definition | | --------------------------- | -------------------------------------------------------------------------------------------------------------- | | `markdown` | The Main Markdown. | | `json` | The State JSON, one entry per piece of evidence. | | `meta.snapshotId` | The capture ID. Pass it to `snapshots.get`. | | `meta.resources.snapshotId` | The public Snapshot ID. Pass it to `snapshots`. See [Two IDs](https://expand.ai/docs/wayback-machine#two-ids). | | `meta.playground` | The Playground link, for a human to open. | | `data.highlights` | Fetch Highlights, when the request had a `highlights`. | | `data.screenshot` | The screenshot's `url`, when the request asked for one. | | `originError` | The origin's HTTP error, when it answered with a status of 400 or above. The content is still there. | > **Note:** > > `isFetchOriginError()` only catches an origin that never answered. Its `kind` names the failure: `dns`, `tls`, `connectionRefused`, `connectionReset`, `connectionClosed`, `addressUnreachable`, `emptyResponse`, or `connectionTimeout`. An origin that answered with an error still returns content, and reports the status in `originError`. The full shape is in the [API reference](https://expand.ai/api-reference/fetch/fetch.fetchJson). ### Markdown Mode with `fetch` `fetch()` calls `/v1/fetch` and returns Markdown text as one string, with the metadata in YAML frontmatter at the top. Use it when the result goes straight into a prompt. ```ts const markdown = await expand.fetch({ url: 'https://www.figma.com/pricing/' }) ``` ### Include options Both methods take an `include` string in their second argument. It is the URL's `?include=` parameter, and it turns State JSON and the Appendix on and off. The request body's `include` does the same with `appendix` and `json` when you leave the string out, and in JSON Mode it also asks for extras outside the document, such as a screenshot. ```ts const withAppendix = await expand.fetch( { url: 'https://www.figma.com/pricing/' }, { include: 'appendix,statejson' }, ) const withScreenshot = await expand.fetchJson( { url: 'https://www.figma.com/pricing/', include: { screenshot: { fullPage: true } } }, { include: 'appendix' }, ) ``` > **Note:** > > `{ include: null }` leaves the parameter off the URL, the same as not passing it. What each token does, and the defaults, are on [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#include-options). ### Batched Fetch `batched()` starts a run and returns its `id`. `getBatched()` polls it. Each result carries its own `status` next to its `data`, so a URL that failed or was cancelled does not hide the ones that worked. ```ts const run = await expand.batched({ urls: ['https://www.figma.com/pricing/', 'https://www.figma.com/about/'], }) let status = await expand.getBatched(run.id, { limit: '10', offset: '0' }) while (status.batchedStatus === 'QUEUED' || status.batchedStatus === 'RUNNING') { await new Promise((resolve) => setTimeout(resolve, 1000)) status = await expand.getBatched(run.id, { limit: '10', offset: '0' }) } console.log(status.results) ``` - `getBatched()` takes `limit`, `offset`, and `include` as strings, because they are URL parameters. - When you leave out `idempotencyKey`, the SDK generates one and reuses it for its own retries. Pass a stable key to make separate calls, or calls after a restart, land on the same run. - Reusing a key with a different payload returns `409 BatchedIdempotencyConflict`. The run's lifecycle, its options, and the result shape are on [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch). ## Highlights ### Fetch Highlights Add `highlights` to a JSON Mode Fetch and the snippets come back under `data.highlights.snippets`, next to the whole page. ```ts const page = await expand.fetchJson({ url: 'https://docs.stripe.com/webhooks', highlights: { query: 'signature verification', maxResults: 5, minScore: 0.6 }, }) if (!isFetchOriginError(page)) { for (const snippet of page.data?.highlights?.snippets ?? []) { console.log(snippet.source, snippet.score, snippet.text) } } ``` Each snippet has `source` (`markdown`, `appendix`, or `statejson`), `text`, `score`, and `location`. A State JSON snippet also has `json`, the matched value. The SDK does not add a `citationUrl` to snippets. Build one with the [citation helpers](https://expand.ai/docs/typescript-sdk#citation-helpers). ### Read a Snapshot ```ts const catalog = await expand.snapshots.get({ snapshotId: '' }) console.log(catalog.links.markdown) ``` This reads the public catalog without sending credentials. Adding `highlights` switches to an authenticated POST and costs one Highlights credit. The return type follows the options you pass. ### Snapshot Highlights `snapshots.get()` with `highlights` calls `/v1/snapshots//highlights` and highlights a Snapshot you already have, without fetching the page again. Pass `meta.snapshotId` from the earlier Fetch. ```ts const page = await expand.fetchJson({ url: 'https://docs.stripe.com/webhooks' }) if (!isFetchOriginError(page)) { const result = await expand.snapshots.get({ snapshotId: page.meta.snapshotId, highlights: { query: 'retry policy', maxResults: 5, minScore: 0.6 }, include: { markdown: true, json: true, appendix: false }, }) console.log(result.highlights.snippets) } ``` The result has `highlights.snippets` in the same shape as above, plus `snapshotId`, `playground`, `response`, and `durationMs`. > **Warning:** > > Without `include`, Highlights uses the Main Markdown and State JSON. When you pass `include`, a layer you leave out is off. Turning all three off returns `400 BadRequest`. How the two ways differ, and how to tune `maxResults` and `minScore`, is on [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight). ## Wayback Machine ### Citation helpers A snippet's `location` points at the exact spot on the Snapshot. The helpers turn it into a link a person can open. ```ts import { citationUrl, resolvePlaygroundHost } from '@expandai/sdk' if (!isFetchOriginError(page)) { const host = resolvePlaygroundHost(page.meta.playground, 'https://expand.land') for (const snippet of page.data?.highlights?.snippets ?? []) { console.log(snippet.text, citationUrl(page.meta.snapshotId, snippet.location, host)) } } ``` | Helper | Definition | | ------------------------------------------------------ | ------------------------------------------------------------------ | | `citationUrl(snapshotId, location, host?)` | The citation link for one snippet, from `location.evidenceId`. | | `playgroundBase(snapshotId, host?)` | The Playground link for the whole Snapshot. | | `playgroundContentUrl(snapshotId, evidenceId?, host?)` | The Main Markdown as text, or one piece of it. | | `playgroundAppendixUrl(snapshotId, host?)` | The Appendix as text. | | `resolvePlaygroundHost(metaPlayground, fallback)` | The host from `meta.playground`, or `fallback` when it is missing. | | `parseSnapshotHandle(handle)` | The `snapshotId` and host from a Snapshot ID or a Playground link. | What each link shows is on [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay#urls). ### Progressive disclosure A Fetch with a saved Snapshot returns `meta.resources`. Its `snapshotId` is the public ID, and the `snapshots` methods read the catalog behind it without an API key and without fetching the page again. ```ts const page = await expand.fetchJson({ url: 'https://www.figma.com/pricing/' }) if (isFetchOriginError(page) || !page.meta.resources) throw new Error('No Snapshot was saved.') const snapshotId = page.meta.resources.snapshotId const catalog = await expand.snapshots.get({ snapshotId }) const network = await expand.snapshots.network({ snapshotId }) const preview = await expand.snapshots.preview({ snapshotId, artifactId: 'html.original', maxBytes: 512 }) const har = catalog.artifacts.find((artifact) => artifact.id === 'har') if (har) { const bytes = await fetch(new URL(har.url, page.meta.resources.url)).then((response) => response.arrayBuffer()) } ``` > **Note:** > > Catalog links are paths on the API origin, so resolve them against `meta.resources.url` before downloading. `snapshots.network()` and `snapshots.preview()` return `unknown`, so narrow the value before using it. What the catalog holds and how an agent reads it is on [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). ## Crawl and WebSync `crawls.create()` starts a crawl. It returns one crawl per registrable domain in your sources, and needs an idempotency key you keep. `crawls(id)` picks up a crawl you started earlier, and `crawls.merge()` reads several as one stream. ```ts const crawls = await expand.crawls.create( { sources: ['https://example.com/products'], formats: ['markdown', 'stateJson'] }, { idempotencyKey: 'catalog-import-2026-09' }, ) for await (const event of expand.crawls.merge(crawls).events()) { // Handle page.added, page.changed, page.removed, page.failed, then persist event.offset. } ``` | Method | Definition | | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `crawl.events({ from?, live?, signal? })` | Stream page events. Pass a saved offset as `from` to resume. `live: true` keeps waiting on a synced crawl. | | `crawl.status()`, `crawl.done()` | Read the crawl's state, or wait until it is done, failed, or canceled. | | `crawl.update({ maxPages?, sync?, changePolicy? })` | Raise limits or change sync settings while the crawl is open. | | `crawl.cancel()` | Stop the crawl. | | `crawl.snapshot()` | Read the current set of pages, then continue `events` from the offset it ends on. | | `crawl.pages()`, `crawl.page(id)`, `crawl.pageByUrl(url)` | Browse the pages the crawl has kept. | | `expand.captures(id).markdown()`, `.html()`, `.stateJson()` | Read a capture's full content when an event's was cut at the 64 KiB cap. | | `expand.pages(id).versions()`, `.diff({ from, to })` | Read a synced page's history, or compare two captures. | Crawl errors are a `CrawlClientApiError` with `code`, `status`, and `retryable`. An offset too old to resume from throws `CrawlOffsetExpiredError`. ### Crawl options | Option | Definition | Default | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | | `sources` | 1 to 1,000 starting points. A URL string, or `{ url, kind }` where `kind` is `'seed'` (follow links from here) or `'sitemap'` (read the URLs it lists). | Required | | `formats` | What each page event carries: any of `markdown`, `stateJson`, `html`. | `['markdown', 'stateJson']` | | `maxPages` | Pages the crawl keeps, up to 1,000,000. | 25,000 | | `maxDepth` (per source) | Clicks from the source URL, 0 to 100. | 50 | | `include` / `exclude` (per source) | Path globs. `include` decides what is kept, and `exclude` keeps the crawl out. | None | | `maxPages` (per source) | A cap for one source. Up to 32 sources may set their own. | None | | `changePolicy` | CSS selectors to ignore when deciding whether a page changed. See [WebSync](https://expand.ai/docs/websync). | None | | `sync` | Keeps the crawl running on a schedule. See [WebSync](https://expand.ai/docs/websync). | None | ### Crawl events Every event has `offset`, `id`, `observedAt`, `crawlId`, and `type`. | Type | Sent when | Fields it adds | | -------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | | `page.added` | A new page was captured. | `pageId`, `url`, `captureId`, `discoveredBy` (`seed`, `sitemap`, `link`, or `search`), `depth`, `meta`, `page` | | `page.changed` | A page changed since its last capture. Sent by WebSync. | `pageId`, `url`, `captureId`, `previousCaptureId`, `changed`, `diff`, `meta`, `page` | | `page.removed` | A page is gone. Sent only after at least three checks agree. | `pageId`, `url`, `previousCaptureId`, `evidence` | | `page.failed` | A page could not be captured. | `url`, `reason`, optional `detail`, `retriesExhausted` | - **`page.changed`** says what changed in `changed` (`content`, `stateJson`, `extraction`) and how in `diff`: changed State JSON paths with their old and new values, and a Markdown diff. - **`page.removed`** says why in `evidence.kind`: `http-404`, `http-410`, `content-gone`, or `redirected-away`, with the checks behind it. - **`page.failed`** gives a `reason`: `blocked`, `not-found`, `timeout`, `error`, `robots-disallowed`, `robots-unreachable`, or `out-of-scope`. It carries no content. Pages left out by `include` never produce one. `event.page` holds the formats you asked for when you created the crawl: `markdown`, `stateJson`, `html`. Content in an event is capped at 64 KiB, filled in this order: State JSON, Markdown, the diff, then HTML. A field that does not fit is left out and named in `event.page.truncated`. The full capture is always available by its `captureId`: **TypeScript SDK** ```ts const markdown = await expand.captures(event.captureId).markdown() ``` **cURL** ```bash curl https://api.expand.ai/v1/captures/$CAPTURE_ID/markdown \ -H "x-expand-api-key: $EXPAND_API_KEY" ``` `event.meta` carries the page's title, description, canonical URL, language, page type, and Open Graph data. What a crawl is, how it finds pages, its states, and its errors are on [Crawl](https://expand.ai/docs/crawl). What a change looks like is on [WebSync](https://expand.ai/docs/websync). ## Errors and retries `ExpandClient` throws one of four errors. Catch `ExpandClientError` to handle any of them. | Error | Definition | | ----------------------------- | ------------------------------------------------------------- | | `ExpandClientError` | The base class. Also thrown for invalid client options. | | `ExpandClientApiError` | The API answered with a 4xx or 5xx. Read `status` and `body`. | | `ExpandClientConnectionError` | The API could not be reached. | | `ExpandClientTimeoutError` | The call ran past `timeoutMs`. | ```ts import { ExpandClientApiError } from '@expandai/sdk' try { await expand.fetchJson({ url: 'https://www.figma.com/pricing/' }) } catch (error) { if (error instanceof ExpandClientApiError) { console.error(error.status, error.body) } throw error } ``` The SDK retries a failed attempt up to `maxRetries` times, with backoff, inside the call's `timeoutMs`: - Timeouts, connection errors, `408`, `409`, `429`, and `5xx` are retried. A `409` is retried unless it is a Batched Fetch conflict, `BatchedIdempotencyConflict` or `BatchedCancellationConflict`, which the SDK never retries. - The `502` page errors (`FetchNavigationFailed`, `FetchCaptureFailed`) and `504 FetchCaptureTimeout` are not. All are verdicts about the page, and a retry would re-run a full capture for the same outcome. A `504` does not prove the page was at fault, so a deliberate manual retry can still make sense. - `529 FetchCapacityTimeout` is retried, honoring its `Retry-After: 5`. What each error means, and whether to retry it, is on [Errors & retries](https://expand.ai/docs/errors-and-retries). ### Choosing `timeoutMs` `timeoutMs` is one budget for the whole call, retries and backoff included, and the SDK enforces it on your side. A Fetch can take up to its [150-second budget](https://expand.ai/docs/fetch/browser-rendered-pages#readiness-and-timing) plus a few seconds to answer, so the `60000` default cuts off long captures with `ExpandClientTimeoutError` before Expand answers. > **Recommendation:** > > Set `timeoutMs` to `180_000` for a single attempt. Add another 180 seconds per retry you allow, or set `maxRetries: 0` when you only want one verdict. ```ts const page = await expand.fetchJson( { url: 'https://www.figma.com/pricing/' }, { timeoutMs: 180_000, maxRetries: 0 }, ) ``` ## Effect `ExpandService` is the same client for applications already built on [Effect](https://effect.website/). It exposes the same methods, with failures typed in the error channel as `ExpandSdkError`, `ExpandApiError`, `ExpandConnectionError`, or `ExpandTimeoutError`. ```ts import { NodeRuntime } from '@effect/platform-node' import { Effect } from 'effect' import { ExpandService, isFetchOriginError } from '@expandai/sdk' const program = Effect.gen(function* () { const expand = yield* ExpandService const page = yield* expand.fetchJson({ url: 'https://www.figma.com/pricing/' }) if (isFetchOriginError(page)) { yield* Effect.logWarning(page.kind, page.retryable) } else { yield* Effect.log(page.markdown) } }) NodeRuntime.runMain(program.pipe(Effect.provide(ExpandService.layer()))) ``` `ExpandService.layer()` takes the same options as `ExpandClient`. You do not need it to use the SDK from plain Promise code. ## Next steps - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url): the two modes, what each layer holds, and the include options. - [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight): Fetch Highlights and Snapshot Highlights, and how to tune them. - [Errors & retries](https://expand.ai/docs/errors-and-retries): what each error means and whether to retry. - [API reference](https://expand.ai/api-reference): the exact request and response schemas. --- # Why Expand (https://expand.ai/docs/why-expand) Your agent reads the HTML. It should see the page. What ordinary Markdown APIs drop, and how Expand keeps it. Your agent reads the HTML. It should see the page. An HTML-to-Markdown converter works from the source and guesses what matters, and the guess is silent. Tables lose their columns, icons vanish, and the JSON a page loads for itself never appears. So the agent gets stuck, or misses the answer, or states something false, and you can't tell which happened. Driving a full browser for every page fixes that, at the cost of speed and money. Expand takes a different route. You send a URL. Expand opens it in a real browser, and our model Iris looks at the rendered page and returns it whole as Markdown, with layout, tables, and icons intact and the trash removed. The page's hidden JSON comes back next to it as [State JSON](https://expand.ai/docs/fetch#state-json). Every line links back to the exact spot on the page, so you can check every result at API speed. ## What ordinary markdown APIs drop HTML-to-Markdown converters decide what counts as content with fixed rules, and the rules fail in three ways. They keep things they shouldn't. They drop things they shouldn't. And some things they cannot see at all. The drops are random and silent, so you never know what your agent didn't see. | What gets dropped | What your agent sees instead | Example | | --------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------- | | Table structure | Cells collapse into loose text and values detach from their column headers. | A pricing page. Which plan has which feature can't be recovered. | | Icons | A checkmark column and a cross column read as identical empty cells. | Any comparison table. Yes and no look the same. | | Script-tag JSON | Data that never rendered as text does not exist. | A chart page. The whole dataset lives in a script tag. | | JavaScript-rendered content | An empty application shell. | Any single-page app fetched without a browser. | | Layout | Reading order scrambles and sidebars interleave with the main content. | Multi-column docs pages. | | Bot-blocked pages | A challenge page or an error, sometimes returned as if it were content. | Retail and news sites behind bot protection. | Expand renders the full page the way a browser does and reads the rendered result, so the data those tools lose survives. Iris looks at the rendered page and decides what to show first. You get a clean [Main Markdown](https://expand.ai/docs/fetch#main-markdown) plus the hidden JSON that other tools never look at. Oversized secondary regions, such as navigation and footers, go in a separate section called the [Appendix](https://expand.ai/docs/fetch#appendix). The goal is more information per token, not shorter Markdown at any cost. ## Completeness versus cost A web page carries more than its main content. There is app state, links, sidebars, and noise such as ads and cookie banners. Most tools pick one of two failure modes. Keeping everything floods the model with low-signal tokens, and you pay for every one of them. Keeping only the main content swings the other way. Pruning to the article body is cheap, but it throws away the links to follow next and the app state that was never in the prose. Expand takes a third path. It looks at the rendered page and the page's internal JSON and selects what to show first. The rest stays one call away. ## Progressive disclosure Progressive disclosure keeps secondary page content available without putting it all in front of your agent. The Main Markdown and State JSON answer most questions. The Appendix, the original HTML and JSON, and the page's assets stay in the [Snapshot](https://expand.ai/docs/fetch#snapshot--evidence), one follow-up call away. How your agent goes back for them is on [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). ## What Expand keeps Expand keeps the icons and layout that make a pricing table legible. It returns State JSON, which answers questions Markdown alone cannot, especially on data-driven pages. Most agents don't read a page's internal JSON today because the whole of it would overwhelm their context window, so Iris selects just the useful fields and elements. Every browser page Fetch is stored as a Snapshot, with the complete HTML and the page's assets. That is how the Playground replays the page as it appeared when you fetched it, and how your agent can go back for an image or a script-tag payload later. The Snapshot is also what makes citation links possible. They are a sourcemap for the extracted web page. Every line of Main Markdown and every State JSON value links back to the exact spot on the original page. Your agent can put a citation link in its answer, and you can click it to see the source highlighted in the Playground. See [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay). ## Why this improves agent answers - Your agent gets the information that is usually dropped, such as hidden JSON, icons, and table layout. - Fewer low-signal tokens, which helps models that degrade as they approach their context limit. - Your agent can go back to the Snapshot for any piece of the page, not only the parts in the Main Markdown. - Every claim links back to the page, so you can check it. ## When Expand is worth it | Use Expand for | A simpler path may be enough | Use browser automation when | | -------------------------------------------- | --------------------------------------- | ------------------------------------------------- | | Agent research | Simple static articles | The task is a long workflow | | Docs ingestion | Markdown that is already clean | The goal is form filling | | Product and pricing pages | A known API exposes the exact data | Extraction is secondary to interaction | | Dashboards and app-like pages | Showing your sources does not matter | Complex interaction determines the result | | Marketplaces, search pages, and detail pages | The page is already structured for LLMs | The agent must operate the page, not only read it | JavaScript-rendered pages, and anything with important tables, icons, layout, or hidden JSON, are where Expand helps most. The same goes for any answer you need to show the source of. When the source is a clean static article or a known API that already returns the exact data, a plain HTTP request or a simpler converter is fine. ## Next steps - [Quickstart](https://expand.ai/docs/quickstart): run one Fetch. - [Quickstart: Ways to use Expand](https://expand.ai/docs/quickstart#ways-to-use-expand): choose your agent, the CLI, the TypeScript SDK, the API, or the Playground. - [Fetch: How Fetch works](https://expand.ai/docs/fetch#how-fetch-works): Main Markdown, State JSON, Appendix, and the Snapshot. - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url): the two modes and the include options. --- # Crawl (https://expand.ai/docs/crawl) Give Crawl a URL and get the pages it finds within your limits, each one a full Fetch, as a stream you can resume. Give Crawl a URL and it fetches the pages it finds within your limits. Each page it keeps is a full Fetch, with [Main Markdown](https://expand.ai/docs/fetch#main-markdown), [State JSON](https://expand.ai/docs/fetch#state-json), and a [Snapshot](https://expand.ai/docs/fetch#snapshot--evidence). Pages arrive one at a time as events on a stream, so you can process them as they come and pick up where you left off. A crawl stays on the URL's registrable domain and its subdomains. A crawl starting at `docs.example.com` can also find pages on `shop.example.com`. ## How a crawl finds pages - **`maxDepth` counts clicks from your URL.** `0` is the URL alone. `1` adds linked pages and pages found through the site's sitemap or search, which count as one click. The default, 50, allows discovery up to `maxPages`. - **A `kind: 'sitemap'` source is the exception.** The URLs it lists start at depth 0, as if each were your URL. - **`include` decides what ends up in the results, not where the crawl goes.** The crawl may open a page that does not match so it can follow its links. It does not keep that page, and the page does not count toward `maxPages`. - **`exclude` keeps the crawl out.** A matching page is never opened, so nothing behind it is found through it. ## Starting a crawl **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const [crawl] = await expand.crawls.create( { sources: ['https://example.com/products'], formats: ['markdown', 'stateJson'] }, { idempotencyKey: 'catalog-import-2026-09' }, ) for await (const event of crawl.events()) { if (event.type !== 'page.added') continue // Process event.page, then persist event.offset. } ``` **CLI** ```bash # Mirrors the site into a folder of Markdown files. The SDK and API stream events instead. expandai crawl https://example.com/products ./products ``` **cURL** ```bash # Start the crawl. The response lists one crawl per domain. curl https://api.expand.ai/v1/crawls \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "x-idempotency-key: catalog-import-2026-09" \ -H "Content-Type: application/json" \ -d '{"sources":["https://example.com/products"],"formats":["markdown","stateJson"]}' # Read its events from the beginning. curl "https://api.expand.ai/v1/crawls/$CRAWL_ID/events?offset=-1" \ -H "x-expand-api-key: $EXPAND_API_KEY" ``` Every request needs an idempotency key. Sending the same key with the same request returns the crawls it already created instead of starting new ones. Sending it with a different request is rejected with `409 idempotency_key_conflict`. **Recommendation:** Persist the idempotency key, so a retry or a restart lands on the crawls you already started. `create` returns a list. Sources are grouped by domain, and each domain becomes its own crawl, so two sites in one request come back as two crawls. `expand.crawls.merge(crawls)` reads their events as one stream. **Warning:** Every source must resolve to a public address. Otherwise the request is rejected with `422 crawl_blocked`. Every option, with its default and limit, is on [TypeScript SDK: Crawl options](https://expand.ai/docs/typescript-sdk#crawl-options). ## What you get back A crawl does not return one big result. It streams events as it finds pages, in order. There are four kinds: | Event | Sent when | | -------------- | ----------------------------------------------------------------------------------------- | | `page.added` | A new page was captured. It carries the page in the formats you asked for. | | `page.changed` | A page changed since its last capture. Sent by [WebSync](https://expand.ai/docs/websync). | | `page.removed` | A page is gone. | | `page.failed` | A page could not be captured. It carries a `reason`. | Every event carries an `offset`. Store it after you have handled the event, and pass it as `from` to pick up where you left off: ```ts for await (const event of crawl.events({ from: savedOffset })) { // ... } ``` **Note:** When you resume, you may see an event you already handled. Use `event.id` to skip duplicates. If an offset is too old to resume from, the stream answers `410 crawl_stream_offset_expired`. Start again from the beginning, or read the current set of pages with `crawl.snapshot()` and continue from the offset it ends on. The content in an event is capped at 64 KiB. A field that does not fit is left out and named in `event.page.truncated`. The full capture is always available by its `captureId`: **TypeScript SDK** ```ts const markdown = await expand.captures(event.captureId).markdown() ``` **cURL** ```bash curl https://api.expand.ai/v1/captures/$CAPTURE_ID/markdown \ -H "x-expand-api-key: $EXPAND_API_KEY" ``` The fields each event carries are on [TypeScript SDK: Crawl events](https://expand.ai/docs/typescript-sdk#crawl-events). ## Crawl states Read a crawl's state with `crawl.status()`, or wait for it to finish with `crawl.done()`. Over HTTP, it is `GET /v1/crawls/:id`. | State | Definition | | ---------- | ---------------------------------------------------------------------------------------------------------- | | `crawling` | Finding and capturing pages. | | `done` | Finished. Nothing is left to find or capture. | | `syncing` | Finished its first pass and is rechecking pages on its [WebSync](https://expand.ai/docs/websync) schedule. | | `paused` | Waiting, for example because your account cannot pay for the next pages. | | `canceled` | Stopped by you. | | `failed` | Could not finish, for example because every source failed and no page was kept. | A crawl without `sync` closes for good when it is done, fails, or is canceled. Create a new crawl to run it again. ## Why a crawl stops early A finished crawl reports whether it covered everything it found. When it did not, `coverage` lists the causes: | Cause | What happened | What to do | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | `max-pages` | More pages were found than `maxPages` allows. | Raise `maxPages` while the crawl is still open. | | `source-limit` | A source hit its own `maxPages`. | Raise that source's cap. | | `source-partial` | A sitemap could not be read in full. | Check that the sitemap is reachable. | | `trap-guard` | URLs that look like crawl traps were skipped: very long URLs, more than 20 query parameters or path segments, or a path segment repeated 3 or more times. | Usually nothing. Add `exclude` patterns if real pages were skipped. | | `origin-errors` | The site returned too many errors in a row, so the crawl backed off. | Try again later. | | `billing` | The crawl paused because your account cannot pay for the next pages. | Add credits. A crawl still paused for billing after 24 hours is canceled. | ## Changing or stopping a crawl While a crawl is open, you can raise `maxPages` or change its sync settings with `crawl.update(...)`, or `PATCH /v1/crawls/:id`. `crawl.cancel()`, or `POST /v1/crawls/:id/cancel`, stops it. Queued pages are released and no more are captured. Canceling a crawl that is already closed returns its status. ## Credit usage Every page Expand renders costs `1` credit, the same as a Fetch. Pages the crawl opens only to follow their links, and never keeps, cost nothing. See [Pricing & Usage](https://expand.ai/docs/pricing-and-usage). ## Crawl errors On top of the [standard errors](https://expand.ai/docs/errors-and-retries), crawl requests use their own shape, `{ error: { code, message, retryable } }`. In the SDK they are a `CrawlClientApiError` with `code`, `status`, and `retryable`. The errors relevant to creating, updating, streaming, and reading a crawl are: | `code` | HTTP | Retryable | | ----------------------------------------------------------------------------------------- | ----- | --------- | | [`idempotency_key_conflict`](https://expand.ai/docs/crawl#idempotency_key_conflict) | `409` | No | | [`crawl_run_conflict`](https://expand.ai/docs/crawl#crawl_run_conflict) | `409` | No | | [`crawl_blocked`](https://expand.ai/docs/crawl#crawl_blocked) | `422` | No | | [`budget_exhausted`](https://expand.ai/docs/crawl#budget_exhausted) | `429` | No | | [`not_found`](https://expand.ai/docs/crawl#not_found) | `404` | No | | [`invalid_offset`](https://expand.ai/docs/crawl#invalid_offset) | `400` | No | | [`crawl_stream_offset_expired`](https://expand.ai/docs/crawl#crawl_stream_offset_expired) | `410` | No | | [`capture_expired`](https://expand.ai/docs/crawl#capture_expired) | `410` | No | | [`temporarily_unavailable`](https://expand.ai/docs/crawl#temporarily_unavailable) | `503` | Yes | ### `idempotency_key_conflict` **HTTP 409 · Retryable: no** The idempotency key was already used for a different request. ```json { "error": { "code": "idempotency_key_conflict", "message": "…", "retryable": false } } ``` Use a new key for a new request. ### `crawl_run_conflict` **HTTP 409 · Retryable: no** A closed crawl cannot start another run, or the run command cannot be replayed. Create a new crawl for a new run. ### `crawl_blocked` **HTTP 422 · Retryable: no** A source does not resolve to a public address. ```json { "error": { "code": "crawl_blocked", "message": "…", "retryable": false } } ``` ### `budget_exhausted` **HTTP 429 · Retryable: no** Your account cannot pay for the crawl. ```json { "error": { "code": "budget_exhausted", "message": "…", "retryable": false } } ``` Add credits, then create the crawl again. ### `not_found` **HTTP 404 · Retryable: no** No crawl with that ID exists for your organization. ```json { "error": { "code": "not_found", "message": "…", "retryable": false } } ``` ### `invalid_offset` **HTTP 400 · Retryable: no** The `from` offset or snapshot cursor is invalid. Use the offset returned by the matching endpoint. ### `crawl_stream_offset_expired` **HTTP 410 · Retryable: no** The requested event history is no longer retained. Replay from offset `-1`, or read the current catalog with `crawl.snapshot()` and resume from its final offset. ### `capture_expired` **HTTP 410 · Retryable: no** A capture ID is beyond its retained horizon. The full content for that capture is no longer available. ### `temporarily_unavailable` **HTTP 503 · Retryable: yes** Expand could not serve the request right now. ```json { "error": { "code": "temporarily_unavailable", "message": "…", "retryable": true } } ``` Try again. The SDK retries it for you. ## Next steps - [WebSync](https://expand.ai/docs/websync): keep a crawl running and get every change as an event. - [TypeScript SDK: Crawl and WebSync](https://expand.ai/docs/typescript-sdk#crawl-and-websync): every option, method, and event field. - [Pricing & Usage](https://expand.ai/docs/pricing-and-usage): what a crawl costs, and how credits are billed. - [Errors & retries](https://expand.ai/docs/errors-and-retries): the standard errors every request can return. --- # Batched Fetch (https://expand.ai/docs/fetch/batched-fetch) Send a list of URLs in one request, then poll one run ID for the results as they finish, without holding a request open per page. A Batched Fetch runs one Fetch for each URL in a list, as one job. You send the URLs in one request and get back a run ID, then poll that ID for each page's Markdown and State JSON as it finishes. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() // Start a run const run = await expand.batched({ urls: ['https://example.com', 'https://example.com/about'], }) // Poll the run ID it returns const page = await expand.getBatched(run.id, { limit: '10', offset: '0' }) ``` **cURL** ```bash # Start a run curl https://api.expand.ai/v1/fetch/batched \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"urls":["https://example.com","https://example.com/about"]}' # Poll the run ID it returns curl "https://api.expand.ai/v1/fetch/batched/019...?limit=10&offset=0" \ -H "x-expand-api-key: $EXPAND_API_KEY" ``` | Use Batched Fetch when | Use single Fetch when | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | You have many URLs with the same settings. | You need one URL now. | | Your pipeline can poll asynchronously. | You need screenshots, the [Appendix](https://expand.ai/docs/fetch#appendix), or [Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight). | | You want results paged from one run ID. | You need everything a single Fetch can return. | **Note:** Batched Fetch is API and SDK only today. There is no CLI command or MCP tool for starting a run. ## Start a run Send all the URLs in one POST. The call returns as soon as the run is accepted. The credit hold and the captures themselves continue in the background. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const run = await expand.batched( { urls: ['https://example.com', 'https://example.com/about'] }, { idempotencyKey: 'acme-catalog:2026-07' }, ) ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch/batched \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "x-idempotency-key: acme-catalog:2026-07" \ -H "Content-Type: application/json" \ -d '{ "urls": [ "https://example.com", "https://example.com/about" ] }' ``` ```json { "id": "019..." } ``` `urls` is required and must be non-empty. `include` is optional, and the only public `browserConfig` field at start is `scrollFullPage`. Unknown top-level keys are rejected. - Every submitted URL uses the same shared settings. The start body does not accept per-URL settings. - Duplicate URLs are deduped by resolved URL, so the same page is captured once. - Production callers should authenticate with an API key. Anonymous sessions cannot start a run. Keep the run ID. Every status check and every page of results is read back through it. **Note:** An immediate poll can return `QUEUED` with no results while the run is starting. This is expected. Credits are checked while the run starts, so a run can later become `FAILED` even though the start request was accepted. ### Safe retries When `x-idempotency-key` is omitted, the API generates a unique key. Send a stable key when the caller needs to retry a separate request or recover after a process restart. Retrying the same payload with the same explicit key is safe. The API returns a successful response with the same run ID and does not create or charge for another run. Only reusing the same key with a different request body returns `409 BatchedIdempotencyConflict`: ```json { "_tag": "BatchedIdempotencyConflict", "existingRunId": "019...", "reason": "payload_mismatch" } ``` **Warning:** Treat `payload_mismatch` as a client bug rather than polling a run created for different input. - Keys are scoped to your organization and may be up to 255 characters. - The API generates a key when omitted. Persist or deterministically reconstruct an explicit key when deduplication must survive separate calls or process restarts. - Include a logical period in recurring-job keys, such as `acme-catalog:2026-07:`. In the TypeScript SDK: - When omitted, `batched()` generates one `idempotencyKey` and reuses it for the client's configured retry policy. - Supply a stable `idempotencyKey` to deduplicate separate calls or calls made after a process restart. - The same key and payload return the same run ID. Catch `ExpandClientApiError` with status `409` only for a different payload. Its body contains `existingRunId` and `reason: "payload_mismatch"`. ## Poll results Read status and results back with the run ID. Page through results with `limit` and `offset`. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const page = await expand.getBatched('019...', { limit: '10', offset: '0' }) ``` **cURL** ```bash curl "https://api.expand.ai/v1/fetch/batched/019...?limit=10&offset=0" \ -H "x-expand-api-key: $EXPAND_API_KEY" ``` Keep polling while `batchedStatus` is `QUEUED` or `RUNNING`. Stop once it reaches a terminal status. ```ts const terminal = new Set(['COMPLETED', 'FAILED', 'CANCELLED']) let status = await expand.getBatched(run.id) while (!terminal.has(status.batchedStatus)) { await new Promise((resolve) => setTimeout(resolve, 1000)) status = await expand.getBatched(run.id) } ``` - Use a short delay or backoff between polls instead of a tight loop. - Handle API errors from the poll call the way you handle any request. - `FAILED` or `CANCELLED` means stop polling the run. - After the run is terminal, inspect each result's `status` before using it. - The generated `getBatched` query params are strings today, so the examples use `"10"` and `"0"`. ### Paging through results Results come back in pages. Start with `offset=0`, then keep increasing `offset` by `limit` while `pagination.hasMore` is true. - `limit` defaults to `10` and accepts `1` through `100`. - `offset` defaults to `0` and must be non-negative. - `pagination.total` is the number of available result rows for the run. - Polling only the first page can miss later results. - Page-level `status` can differ across pages while the run is still active. ## Statuses `batchedStatus` describes the whole run. The top-level `status` describes the current page of results, and each result has its own persisted `status`. | Field | Definition | How to use it | | ------------------ | -------------------------------------------------- | ----------------------------------------------------------------------------- | | `batchedStatus` | Overall run status. | Decide whether to keep polling the run. | | `status` | Current page status, including extraction quality. | Decide whether this page is stable and its requested content was extracted. | | `results[].status` | Persisted state for one URL. | Handle successful, failed, queued, running, and cancelled URLs independently. | ```txt batchedStatus = whole run status = current page ``` - `QUEUED` means the run was accepted but may not be executing yet. - `RUNNING` means work is still in progress. - `COMPLETED` on `batchedStatus` means the run is done, not that every URL produced Markdown. - `FAILED` and `CANCELLED` are terminal. - A page reads `COMPLETED` only when its rows are stable and requested content extraction succeeded. Requested content can legitimately be empty. - A page can read `FAILED` after a completed capture if Snapshot content could not be read or rendered. Successful siblings on that page remain usable. - Failed extraction outcomes are cached for one minute. Re-poll the same failed page with backoff to retry extraction without starting or charging for another browser capture. - A page can read `COMPLETED` while the whole run is still `RUNNING`. Keep the two fields separate in your code. Do not collapse them into one `status`. ## Response shape A poll returns the run status, pagination metadata, and a page of per-URL results. ```json { "id": "019...", "status": "COMPLETED", "batchedStatus": "RUNNING", "totalUrls": 20, "pagination": { "total": 12, "limit": 10, "offset": 0, "hasMore": true }, "results": [ { "status": "SUCCEEDED", "data": { "response": { "url": "https://example.com", "originStatusCode": 200 }, "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples." } }, { "status": "SUCCEEDED", "data": { "response": { "url": "https://blocked.example", "originStatusCode": 403 }, "blocked": { "blockedType": "botProtection" } } }, { "status": "SUCCEEDED", "error": { "_tag": "FetchOriginError", "url": "https://checkout.deadstartup.example/pricing", "kind": "dns", "upstreamError": "net::ERR_NAME_NOT_RESOLVED", "retryable": false } }, { "status": "SUCCEEDED", "data": { "response": { "url": "https://site.example/removed-page", "originStatusCode": 404 }, "markdown": "# Page not found\n\n…" }, "originError": { "_tag": "FetchOriginError", "url": "https://site.example/removed-page", "kind": "http", "originStatusCode": 404, "retryable": false } }, { "status": "FAILED", "data": { "response": { "url": "https://unavailable.example" } } } ] } ``` Handle each result independently. A run can finish even when one URL was blocked, failed, or redirected. Use `results[].status` rather than missing Markdown to identify failed items. A blocked result remains `SUCCEEDED` and carries `data.blocked`. Fully populated siblings remain usable. How to read a failed item, and which to re-run, is in [`error` and `originError`](https://expand.ai/docs/fetch/batched-fetch#error-and-originerror). If no poll `include.markdown` is provided, Batched Fetch normalizes results to Markdown by default. For the exact per-result schema, see the [API reference](https://expand.ai/api-reference/fetch/fetch.getBatched). ### `error` and `originError` An item carries **exactly one of** `data` or `error`. `originError` sits outside that pair and can accompany the captured `data`. | Field | Definition | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `data` | The captured result. Present whenever a Snapshot was produced. | | `error` | Present when this URL produced no snapshot at all. Carries what the synchronous endpoint would have returned: `FetchBlocked`, `FetchPageTooLarge`, `FetchNavigationFailed`, or `FetchOriginError`. | | `originError` | What the *origin* did. Present when the origin responded with a status of `400` or above, alongside the captured `data`. | A URL whose origin could not be reached is not a run failure. That covers a DNS failure, broken TLS, a refused connection, or a connection timeout. The item reads `SUCCEEDED`, because Expand did its job and established that the URL has nothing to serve, and carries a `FetchOriginError` in `error` with no `data`. An origin that answered with a `404` or `500` did produce a Snapshot, so that item keeps its `data` and reports the status in `originError` instead. Batched Fetch is item-oriented. Treat the run as a container for many Fetch attempts, then inspect each result before using it. - Blocked target pages can appear as per-item `data.blocked`, for example `blockedType: "botProtection"`. - An unreachable origin appears as a per-item `error` with `_tag: "FetchOriginError"` and no `data`. Read its `retryable` field before scheduling a re-run: `false` means the URL is dead, not that the run went wrong. - `data.response.originStatusCode` helps classify each outcome, for example `403` on a blocked page. - Pending rows can initially return only minimal `data.response`. - A URL-only, unblocked result on a `FAILED` page can indicate an extraction failure. Missing Markdown on a `COMPLETED` page can be legitimate empty content. - Terminal `COMPLETED` on the run does not override a page-level extraction failure. - Store per-URL outcome state so a retry collects only the URLs that need it. What `kind` and `retryable` mean, and what is charged, is on [Errors & retries](https://expand.ai/docs/errors-and-retries). ## Supported options | Capability | Batched Fetch support | Notes | | -------------------------------------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Markdown | Supported | Always requested and returned by default when capture succeeds. | | HTML | Supported on completed results | Request through `include`. | | Meta | Supported | Included by default when available. | | Links | Supported on completed results | Request through `include.links`. | | Response info | Supported | Each result carries `data.response` with the final URL, requested URL, and status codes. Poll with `include.response.includeHeaders: true` to get the headers too. | | `browserConfig.scrollFullPage` | Supported at start | Applies to every child fetch. | | Screenshots | Not supported in batched results | Use single Fetch. | | [State JSON](https://expand.ai/docs/fetch#state-json) / `json` | Supported | Always requested and returned by default as `data.json` when extraction succeeds. | | Appendix | Not supported in batched results | Use single Fetch. | | Highlights | Not supported | Use [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight) on a single Fetch or existing Snapshot. | | Documents | Not supported | Every URL in a run goes to the browser, so a PDF, image, office file, ebook, or audio URL comes back as a failed item. Use single Fetch. See [PDFs and other files](https://expand.ai/docs/fetch/documents). | | `documentConfig` | Accepted, no effect | Reserved for when batched document conversion lands. | Polling rejects `screenshot` and `appendix` include fields with `BatchedIncludeUnsupported` when the original run did not capture them. New runs always request Markdown and State JSON. Existing runs retain the formats captured when they were created. Batched Fetch is narrower than single Fetch by design. Use it when many URLs and asynchronous collection matter more than every single Fetch artifact. For the full set of fields and nested defaults, see [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#include-options). ## Concurrency and cancellation - Batched Fetch queues its child browser captures under the organization's browser concurrency rather than cancelling each other. - Cancellation is cooperative. A successful cancel request returns `CANCELLING` while running items drain, or `CANCELLED` when no work remains. Queued items are cancelled immediately. - How much runs at once depends on your browser concurrency. See [Limits](https://expand.ai/docs/limits#concurrency-and-rate). - There is no Batched-specific public rate-limit number to plan against today. ## Common mistakes - Polling once, seeing no results, and assuming the run failed. - Stopping because a page `status` is `COMPLETED` while `batchedStatus` is still `RUNNING`. - Ignoring pagination and reading only the first page. - Assuming `COMPLETED` means every item has Markdown. - Re-running URLs whose `error` reports `retryable: false`. The origin is unreachable. The run did nothing wrong. - Requesting `screenshot` or `appendix` while polling when the original run did not capture it. - Looking for a CLI or MCP command for Batched Fetch. There is none today. - Expecting an automatically generated key to deduplicate separate calls or calls made after a process restart. - Treating `payload_mismatch` as a recoverable duplicate instead of fixing key reuse. ## Next steps - [Start Batched Fetch API reference](https://expand.ai/api-reference/fetch/fetch.batched): exact create schema. - [Get Batched Status](https://expand.ai/api-reference/fetch/fetch.getBatched): the exact poll and result schema in the API reference. - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#include-options): the full include surface and limits. - [Browser-rendered pages](https://expand.ai/docs/fetch/browser-rendered-pages): capture and `scrollFullPage` behavior. --- # Browser-rendered pages (https://expand.ai/docs/fetch/browser-rendered-pages) Fetch opens each page in a real browser and runs its JavaScript before writing the Markdown, State JSON, and a Snapshot you can replay. Fetch opens every page in a real browser and runs its JavaScript before reading it. That makes the difference on pages where the raw HTML misses what you need: - docs that load their content with JavaScript - dashboards and apps whose data lives in JSON - pricing pages, tables, and grids where the layout carries meaning - pages where the icons are the signal **Note:** Fetch reads pages. It doesn't fill in forms, click through steps, or drive the page. For that, use browser automation. ## Readiness and timing Fetch decides when the page is ready. There's no option to set a timeout, a wait selector, network idle, or other readiness rules. What to expect: - Pages are rendered before extraction, not captured as raw source. - A page that never settles may be captured partially. - Captures that fail or are refused surface as errors or blocked outcomes. - When output looks off, replay shows what Fetch actually saw. Every Fetch that waits for its result has one **150-second budget**. It starts when the request is accepted and covers everything after that: queueing, loading, settling, scrolling, the Snapshot, and delivery. Retries inside Expand don't reset it. - If the capture had started and the budget runs out, you get [`504 FetchCaptureTimeout`](https://expand.ai/api-reference/errors#capture-timeouts-504). The TypeScript SDK doesn't retry it, so an outage on our side isn't amplified. A manual retry can still help. The timeout doesn't prove the page was at fault. - If the request was still waiting for a browser, you get [`529 FetchCapacityTimeout`](https://expand.ai/api-reference/errors#capacity-timeouts-529) with `Retry-After: 5`. It's a capacity problem, not a page problem. The TypeScript SDK retries it. - Expand takes up to 5 seconds to decide which, so the error arrives by 155 seconds. A result that lands in that window is still returned. - Set your client timeout to **180 seconds**. That leaves 25 seconds for the error to reach you. The TypeScript SDK's `timeoutMs` defaults to 60 seconds and cuts off long captures, so raise it. See [TypeScript SDK](https://expand.ai/docs/typescript-sdk#errors-and-retries). - In a [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch), each item gets its own 150 seconds once it's scheduled, with the same two errors. Preparing the run doesn't count against it, since you poll for results instead of holding one request open. - Fetches from the [Playground](https://expand.ai/docs/wayback-machine/playground-and-replay) follow the same budget and rules. Streamed requests report the outcome in the event stream. **Note:** 150 seconds is the ceiling, not a target. Most captures finish in a few seconds. On a `504`, `timeoutMs` reports the ceiling, `150000` by default. The timings inside the budget, such as readiness waits, are internal and can change. ## Scrolling `scrollFullPage` is the one public browser behavior control. It defaults to `false`. When `true`, Fetch attempts to scroll the page before capture to trigger lazy-loaded content, then returns to the top before capturing. Use `scrollFullPage` when important content appears only after scrolling. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://stripe.com/pricing', browserConfig: { scrollFullPage: true }, }) ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://stripe.com/pricing", "browserConfig": { "scrollFullPage": true } }' ``` **Note:** The scroll pass is best-effort. It does not guarantee that every infinite-scroll page is fully exhausted. A scrolling failure is logged and does not fail the request. ## Screenshots Fetch can take a full-page or viewport screenshot of any page it opens in a browser. To get one, set `include.screenshot` in the request body, not `browserConfig`. It comes back in the JSON object as `data.screenshot`, with its `url` and a `fullPage` flag. Markdown Mode can't include the screenshot, but it's still saved with the Snapshot. JSON Mode returns a link to it. See [Two modes](https://expand.ai/docs/fetch/fetching-a-url#two-modes). Request a full-page screenshot in JSON Mode: **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const result = await expand.fetchJson({ url: 'https://example.com', include: { screenshot: true }, }) ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch/json \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com","include":{"screenshot":true}}' ``` For the visible viewport only, use `include: { screenshot: { fullPage: false } }`. Leave `include.screenshot` out when you do not need an image. **Note:** Asking for a screenshot always uses the browser, even for a PDF or other file that would otherwise skip it. The full schema is in the [API reference](https://expand.ai/api-reference/fetch/fetch.fetchJson). Everything else you can include, and how each mode returns it, is on [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#include-options). ## Redirects Fetch follows redirects and captures the page it lands on. Each mode tells you where it landed in its own way: **JSON object** ```jsonc { "meta": { "url": "https://www.example.com/" // where Fetch landed } } ``` **Markdown text** ```md --- url: http://example.com # what you asked for finalUrl: https://www.example.com/ # where Fetch landed, only after a redirect --- ``` **Note:** A single Fetch returns only the final URL. The rest of the HTTP exchange, such as the origin's status code, the redirect's status code, and the response headers, is returned by Batched Fetch, per item, under `data.response`. ## Blocked pages Fetch uses a real browser, so it gets through many sites that block plain HTTP requests. Fetch does not guarantee to get past every CAPTCHA, login wall, or bot check. - A blocked page returns [`403 FetchBlocked`](https://expand.ai/api-reference/errors#access-and-blocked-errors-403). Its `blockedType` says why: `captcha`, `botProtection`, `ipBlock`, or `authWall`. [You are not charged for blocked requests](https://expand.ai/docs/pricing-and-usage#what-you-are-charged-for-by-outcome). ```json { "_tag": "FetchBlocked", "url": "https://example.com", "blockedType": "botProtection" } ``` - Fetch retries some blocks on its own before giving up. - How Fetch routes traffic is internal. There's no proxy option. - When a result looks wrong, open the replay before assuming the page is unreachable. See [Debug with replay](https://expand.ai/docs/fetch/browser-rendered-pages#debug-with-replay). A capture that was cut short can still return content. In the JSON object, `meta.blocked` or `meta.loadingStatus` marks it as partial. In a Batched Fetch, each item reports its own blocked state. ## What you can and can't configure You can set three browser options. Fetch manages everything else. | Option | Field | Definition | | --------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Scrolling | `scrollFullPage` | Best-effort scroll before capture. See [Scrolling](https://expand.ai/docs/fetch/browser-rendered-pages#scrolling). | | Screenshot | `include.screenshot` | Captures a screenshot. See [Screenshots](https://expand.ai/docs/fetch/browser-rendered-pages#screenshots). | | Screenshot size | `include.screenshot.fullPage` | Defaults to full page if not set. Set to `false` for the viewport only. | These are managed by Fetch and aren't options today: | Control | Definition | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Timeout | Readiness is managed internally. Synchronous Fetch enforces one 150 s budget per request. Batched items get their own when scheduled. Internal retries never reset it. | | Proxy / location | Proxy retry behavior is internal. | | Viewport / device | Not a public Fetch option. | | User agent | Not a public Fetch option. | | Cookies / session auth | Not a public Fetch option. You can't send cookies or sign in. | | Geolocation | Not a public Fetch option. | | Wait selector / action steps | Not a public Fetch option. Fetch reads pages and doesn't drive them. | | Bot detector bypass | Not a public Fetch option. Fetch retries some blocks on its own. | If any of these become public, this page and the [API reference](https://expand.ai/api-reference) change together. ## Debug with replay When output looks incomplete or surprising, open the replay. It shows what Fetch saw at capture time. - Check for missing or lazy-loaded content. - Inspect citations and evidence. - Tell page behavior (what the site rendered) apart from extraction behavior (what Fetch kept). Look for content that loaded late, a capture that was cut short, or an unexpected redirect. [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay) covers the full workflow. ## Next steps - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#include-options): control which artifacts Fetch returns. - [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay): inspect evidence visually. - [How Fetch works](https://expand.ai/docs/fetch#how-fetch-works): the capture mental model. --- # PDFs and other files (https://expand.ai/docs/fetch/documents) Fetch a PDF, image, office file, ebook, data file, or audio file, and get the same agent-ready Markdown back. Not every URL an agent follows is a web page. Research trails end in PDFs, data links end in spreadsheets, and a citation chain will happily drop you on a scanned report. Fetch converts those files on the endpoint you already call and returns the same Markdown. There is nothing to turn on. `POST /v1/fetch` decides for itself whether a URL is a page to render or a file to convert. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://arxiv.org/pdf/1706.03762' }) ``` **CLI** ```bash expandai fetch https://arxiv.org/pdf/1706.03762 ``` **MCP** ```jsonc // Arguments for the fetch tool { "url": "https://arxiv.org/pdf/1706.03762" } ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://arxiv.org/pdf/1706.03762"}' ``` ## What Fetch converts | Family | Formats | How it converts | Extra credits | | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ---------------------------------- | | PDF | `.pdf` | From the text layer. Pages with no usable text go to a vision model. | Per page, plus vision-model pages. | | Images | `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.avif`, `.bmp`, `.tif`, `.tiff` | A vision model reads the text and describes the image. | One vision-model page. | | Office | `.doc`, `.docx`, `.docm`, `.ppt`, `.pptx`, `.pptm`, `.pps`, `.ppsx`, `.ppsm`, `.pot`, `.xls`, `.xlsx`, `.xlsm`, `.xlsb`, `.odt`, `.ods`, `.odp`, `.rtf` | Locally. Spreadsheets become Markdown tables with their number formats intact. | None | | Ebooks | `.epub` | Locally, chapter by chapter. | None | | Data | `.csv`, `.tsv`, `.json`, `.xml`, `.rdf` | Locally. CSV and TSV become tables. JSON and XML keep their structure. | None | | Text | `.txt`, `.text`, `.md`, `.markdown` | Markdown comes back as is. Plain text is fenced so its whitespace survives. | None | | Audio | `.mp3`, `.wav`, `.m4a`, `.aac`, `.ogg`, `.oga`, `.opus`, `.flac` | Transcribed with a speech model. | Per minute. | Everything else, including HTML and SVG, is rendered in a browser and captured as a page. See [Browser-rendered pages](https://expand.ai/docs/fetch/browser-rendered-pages). A YouTube video is its own case and comes back as a transcript. See [YouTube videos](https://expand.ai/docs/fetch/youtube). ### URLs without an extension The extension is only a hint. Fetch looks at what the origin serves, so `https://arxiv.org/pdf/1706.03762` converts as a PDF even though the URL has no extension, and a `.pdf` URL that serves a web page is captured as a page. A spreadsheet served as `application/octet-stream` still converts. **Note:** A URL with no recognizable extension that serves `text/plain` is captured as a page. Give it a `.txt` or `.md` extension, or serve `text/markdown`, to get the text back as is. ## What comes back A file comes back in either of Fetch's two modes, as Markdown text or as a JSON object. See [Two modes](https://expand.ai/docs/fetch/fetching-a-url#two-modes). ### Markdown Mode Markdown Mode returns the converted file as Markdown text. It is the document itself, with no frontmatter added. Details such as the content type and the link to the original file are only in the JSON object. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://arxiv.org/pdf/1706.03762' }) ``` **CLI** ```bash expandai fetch https://arxiv.org/pdf/1706.03762 ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://arxiv.org/pdf/1706.03762"}' ``` ```markdown # Attention Is All You Need ## Abstract The dominant sequence transduction models are based on complex recurrent or convolutional neural networks ... ``` ### JSON Mode JSON Mode returns the converted file as a JSON object. It has the standard fields, plus `data.document`, a link to the original file, and `data.usage`, what the conversion cost. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const page = await expand.fetchJson({ url: 'https://arxiv.org/pdf/1706.03762' }) ``` **CLI** ```bash expandai fetch https://arxiv.org/pdf/1706.03762 --format json ``` **MCP** ```jsonc // Arguments for the fetch tool { "url": "https://arxiv.org/pdf/1706.03762" } ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch/json \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://arxiv.org/pdf/1706.03762"}' ``` ```json { "meta": { "version": 1, "url": "https://arxiv.org/pdf/1706.03762", "capturedAt": "2026-09-01T00:00:00.000Z", "snapshotId": "0f9c3a1e-...", "contentType": "application/pdf" }, "markdown": "# Attention Is All You Need\n\n...", "json": [], "data": { "document": { "url": "https://s3.amazonaws.com/.../document/original.pdf?X-Amz-Signature=...", "contentType": "application/pdf", "sizeBytes": 2215244 }, "usage": { "documentPages": 15, "ocrPages": 0, "audioSeconds": 0, "videos": 0 } } } ``` | Field | Definition | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `markdown` | The whole converted document. | | `meta.contentType` | The detected content type. It can differ from what the origin's header said. | | `meta.snapshotId` | The ID of the stored conversion. There is no page to replay. | | `data.document.url` | A presigned link to the original file, for sharing or downloading it. | | `data.document.sizeBytes` | The size of the original, as downloaded. | | `data.usage` | The pages converted, pages read by the vision model, and seconds of audio transcribed. See [Additional credit usage](https://expand.ai/docs/fetch/documents#additional-credit-usage). | **Note:** A file isn't rendered in a browser, so it has no State JSON, Appendix, links, screenshot, or replay. Asking for them doesn't cause an error. `json` comes back as an empty list, and the other fields are left out. The [Playground](https://expand.ai/docs/wayback-machine/playground-and-replay) previews the converted document instead. ## Additional credit usage Fetching a file costs the usual `1` credit, plus credits for the work of converting it: | Unit | Credits | When it applies | | ----------------- | -------------------------------- | -------------------------------------------------------------- | | Converted page | `1` per page | Every page a conversion delivers. An image counts as one page. | | Vision-model page | `5` per page, on top of the page | A page read by the vision model instead of from a text layer. | | Audio | `10` per minute | Audio files. Rounded up, one-minute minimum. | Office files, ebooks, data files, and text convert locally and add nothing. Those Fetches cost the flat `1` credit. For example, a 12-page PDF with a text layer costs `13` credits. The same document scanned, with no text layer, costs `73` credits. You are only charged for a delivered result, never when a file can't be converted. `data.usage` reports the units each Fetch spent, so you can check how it was charged. See [Pricing & Usage](https://expand.ai/docs/pricing-and-usage#what-you-are-charged-for-by-outcome). ## Limiting what gets converted `documentConfig` caps the work a conversion can do, and so what it can cost. Every field is optional. | Field | Definition | Default | Range | | ----------------- | --------------------------------------------------------------------- | -------- | ------------------------- | | `maxPages` | Refuse the Fetch rather than convert a PDF with more pages than this. | No limit | Any positive integer | | `maxAudioSeconds` | Refuse the Fetch rather than transcribe audio longer than this. | `3600` | `1` to `14400` | | `ocr` | Whether pages may go to the vision model. | `auto` | `auto`, `never`, `always` | The `ocr` modes: - `auto` keeps the readable text from each page and sends pages with missing or unreliable text to the vision model. - `never` keeps the vision model out. It is the cheapest and most predictable setting. It also returns an empty document for a scanned PDF, and nothing at all for an image, since an image has no text layer to fall back on. - `always` reads every page with the vision model, even where a text layer exists. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://www.irs.gov/pub/irs-pdf/fw4.pdf', documentConfig: { maxPages: 10, ocr: 'never' }, }) ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://www.irs.gov/pub/irs-pdf/fw4.pdf", "documentConfig": { "maxPages": 10, "ocr": "never" } }' ``` **Warning:** A file over one of these limits is refused with `413 FetchPageTooLarge`, not cut short. A half-converted document is worse than a clear error, because nothing in the Markdown says where it stopped. A refused Fetch costs nothing. A conversion that is within the limits can still take a while. **Note:** Conversion can run for up to 55 minutes once it starts. Only the wait for a worker uses the [150-second budget](https://expand.ai/docs/fetch/browser-rendered-pages#readiness-and-timing). Set your HTTP client timeout to allow a long conversion to finish. Downloads also have a size cap, and image and audio conversion depend on their providers' limits. ## Batched Fetch [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch) doesn't convert files yet. It renders every URL in a browser, so a file URL comes back as a failed item rather than as Markdown. The batched endpoint accepts `documentConfig` but ignores it. Send file URLs to `POST /v1/fetch` or `POST /v1/fetch/json` until this lands. ## Document errors On top of the [standard Fetch errors](https://expand.ai/docs/errors-and-retries), a file Fetch can return four more errors. Their full payloads are in the [API reference](https://expand.ai/api-reference/errors#document-and-video-errors-404-410-413-415-422-502-503). | `_tag` | Code | Retryable | | ----------------------------------------------------------------------------------------------------------------- | ---------------------- | --------- | | [`FetchUnsupportedContentType`](https://expand.ai/docs/fetch/documents#fetchunsupportedcontenttype) | `415` | No | | [`FetchOriginHttpError`](https://expand.ai/docs/fetch/documents#fetchoriginhttperror) | `404`, `410`, or `502` | Sometimes | | [`FetchDocumentProcessingUnavailable`](https://expand.ai/docs/fetch/documents#fetchdocumentprocessingunavailable) | `503` | Yes | | [`FetchPageTooLarge`](https://expand.ai/docs/fetch/documents#fetchpagetoolarge) | `413` | No | ### `FetchUnsupportedContentType` **HTTP 415 · Retryable: no** The file can't be converted. Its format is unknown, or it is too damaged to read. ```json { "_tag": "FetchUnsupportedContentType", "url": "https://example.com/archive.7z", "contentType": "application/x-7z-compressed" } ``` `contentType` is present when Fetch could detect one. ### `FetchOriginHttpError` **HTTP 404, 410, or 502 · Retryable: sometimes** The origin refused to serve the file. `statusCode` is the status it answered with. ```json { "_tag": "FetchOriginHttpError", "url": "https://example.com/report.pdf", "statusCode": 404 } ``` A missing or removed file comes back as `404` or `410`, and any other refusal as `502`. Only the `502` is worth retrying, and the SDK retries it for you. A page Fetch works differently. When a page's origin answers with an error, Fetch renders that error page and reports the status beside it, because an error page is often the answer you needed. A file has no page to render, so the origin's status comes back as the error. **Note:** An upcoming change reports these refusals as `200 FetchOriginError` with `contentAvailable: false`, `originStatusCode`, and a `retryable` verdict. Support both shapes during the transition. ### `FetchDocumentProcessingUnavailable` **HTTP 503 · Retryable: yes** The vision model or the speech model couldn't finish. ```json { "_tag": "FetchDocumentProcessingUnavailable", "url": "https://example.com/scan.pdf", "operation": "ocr", "message": "OCR provider unavailable" } ``` The `operation` field is set to `ocr` when the vision model couldn't finish, and `transcription` when the speech model couldn't finish. The SDK retries this error automatically, with backoff. ### `FetchPageTooLarge` **HTTP 413 · Retryable: no** The file is over one of your `documentConfig` limits. ```json { "_tag": "FetchPageTooLarge", "url": "https://example.com/book.pdf", "limitKind": "document_pages", "limit": 50, "measured": 312 } ``` `limitKind` says which limit: `document_pages` for `maxPages`, or `audio_seconds` for `maxAudioSeconds`. `limit` and `measured` are in that unit. To convert the file, raise the limit. See [Limiting what gets converted](https://expand.ai/docs/fetch/documents#limiting-what-gets-converted). ## Next steps - [YouTube videos](https://expand.ai/docs/fetch/youtube): video transcripts on the same endpoint. - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#include-options): choose what a Fetch returns. - [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay): inspect a converted document next to its original. - [Pricing & Usage](https://expand.ai/docs/pricing-and-usage): what converted pages and audio cost in credits. - [Errors & retries](https://expand.ai/docs/errors-and-retries): which errors to retry, and what you are charged for. --- # Fetching a URL (https://expand.ai/docs/fetch/fetching-a-url) The two modes, what to include in a Fetch, and the pointers to everything held back. Fetch has two modes. Markdown Mode returns Markdown text, for agents. JSON Mode returns a JSON object, for applications. Both carry the same layers. The Markdown text puts the metadata in YAML frontmatter at the top, and the JSON object puts each layer in its own field. Markdown Mode is the default. It gives your agent the [Main Markdown](https://expand.ai/docs/fetch#main-markdown), enough to work from without flooding its context, and suits first-pass reading, summarization, extraction, and browsing. Include more [layers](https://expand.ai/docs/fetch#how-fetch-works) only when the task needs more context or a different artifact. See [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). ## Two modes The same Fetch in both modes, with every layer included. | Mode | Endpoint | Shape | | ------------- | --------------------- | ------------------------------------------------------------------------------------ | | Markdown Mode | `POST /v1/fetch` | One Markdown document with YAML frontmatter, Main Markdown, and any included layers. | | JSON Mode | `POST /v1/fetch/json` | An object with `markdown`, `json`, `meta`, and requested artifacts. | The Appendix is appended under `## Appendix` in Markdown Mode. In JSON Mode, it is appended to `markdown`. State JSON appears under `## Expand State JSON` in Markdown Mode and in `json` in JSON Mode. **Markdown Mode response**, shortened to show the layers: ````markdown --- url: https://news.ycombinator.com snapshotId: "019..." playground: https://expand.land/s/019... title: Hacker News appendix: url: https://expand.land/s/019.../appendix --- # Hacker News 1. Example story title ## Appendix - [new](/newest) ## Expand State JSON ```json id=3 { "stories": [{ "title": "Example story title" }] } ``` ```` **JSON Mode response**, shortened to show the same layers: ```json { "meta": { "url": "https://news.ycombinator.com", "snapshotId": "019...", "playground": "https://expand.land/s/019...", "title": "Hacker News", "appendix": { "url": "https://expand.land/s/019.../appendix" } }, "markdown": "# Hacker News\n\n1. Example story title\n\n## Appendix\n\n- [new](/newest)", "json": [ { "id": 3, "kind": "inline", "role": "ld+json", "body": { "stories": [{ "title": "Example story title" }] } } ] } ``` These are illustrative responses. The [API reference](https://expand.ai/api-reference/fetch/fetch.fetchJson) defines all metadata and response fields. Pick Markdown Mode when the response goes straight to an agent or a prompt. The Markdown text is one string, ready to read. Pick JSON Mode when your code reads the fields separately, such as the `snapshotId` or the [State JSON](https://expand.ai/docs/fetch#state-json). **Note:** Links and images on the page's own site come back relative, such as `/pricing`, or `#faq` for the page itself. Resolve them against `finalUrl`, or `url` when there is no `finalUrl`. Links to other sites stay absolute. The Main Markdown always comes back whole, however long the page. Only a [Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight) request returns less. Citations point back to the page through evidence IDs: `json[].id` on State JSON entries, and `location.evidenceId` on Highlights results, which come back in `data.highlights`. ## Include options You choose what a Fetch returns in two places, in either mode: - **The URL's `?include=` parameter** turns two layers on and off: State JSON and the [Appendix](https://expand.ai/docs/fetch#appendix). An empty `?include=` turns State JSON off. `?include=appendix,statejson` turns both on. - **The request body's `include.appendix` and `include.json`** do the same when the URL has no `?include=`. Without either, you get State JSON but not the Appendix. - **The rest of the body's `include` field** asks for extra artifacts outside the document, such as a screenshot. Only JSON Mode can return one. In Markdown Mode, a requested screenshot is saved with the Snapshot but not returned. | Output | TypeScript SDK and HTTP API | CLI | MCP `fetch` | | -------------- | --------------------------------------------------------------- | ---------------------------------------------------- | -------------------------------------- | | Main Markdown | Always returned | Always returned | Always returned | | State JSON | On by default; URL `include` or body `include.json` | On; no layer flag | `includeJson`, defaults to `true` | | Appendix | Off by default; URL `include` or body `include.appendix` | Read afterward with `expandai appendix ` | `includeAppendix`, defaults to `false` | | Page meta tags | On by default; JSON Mode can disable with `include.meta: false` | On | `includeMeta`, defaults to `false` | | Screenshot | JSON Mode, `include.screenshot` | Not available | Not available | The CLI selects JSON Mode with `--format json`. MCP always returns a JSON object. Core metadata such as the Snapshot ID is separate from the optional page meta tags. For example, this JSON Mode request includes State JSON, the Appendix, and a screenshot: **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const page = await expand.fetchJson( { url: 'https://news.ycombinator.com', include: { screenshot: true } }, { include: 'appendix,statejson' }, ) ``` **CLI**, using its fixed layer defaults: ```bash expandai fetch https://news.ycombinator.com --format json expandai appendix ``` **MCP**, including the Appendix and page meta tags: ```jsonc // Arguments for the fetch tool. Screenshots are not available in MCP. { "url": "https://news.ycombinator.com", "includeJson": true, "includeAppendix": true, "includeMeta": true } ``` In the JSON response, the Appendix is added to `markdown`, State JSON entries appear in `json`, and a requested screenshot appears under `data.screenshot` with `url` and `fullPage`. A screenshot is saved but not returned in Markdown Mode. **cURL** ```bash curl 'https://api.expand.ai/v1/fetch/json?include=appendix,statejson' \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://news.ycombinator.com","include":{"screenshot":true}}' ``` Leave out `?include=` to use the defaults. An empty `?include=` turns State JSON off. Rules to remember: - Use `statejson` in URLs. Other spellings, such as `stateJson` and `state-json`, also work. - Unknown tokens are ignored. - `?include=appendix` on its own turns State JSON off. List both to keep both. - When the URL has `?include=`, even an empty one, it wins over the body's `include.appendix` and `include.json`. - A screenshot can cover the whole page. See [Browser-rendered pages: Screenshots](https://expand.ai/docs/fetch/browser-rendered-pages#screenshots). - In a [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch#supported-options), set options in the start request's body. Polling takes an `include` too, but only for what the run captured. ## References to more To keep responses small, Fetch sends a pointer instead of content that's large or secondary. Each pointer says what was held back and where to get it, so nothing is lost. | Pointer | What was held back | How to get it | | ----------------------------------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `appendix` in the frontmatter, `meta.appendix` in the JSON object | The Appendix | Open its `url`, or fetch with `?include=appendix` | | `[appendix omitted: …] {612}` in the Main Markdown | One piece of the Appendix | `expandai appendix --id 612` | | `stateJson.moreTokens` | Some State JSON, about that many tokens | Run [Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight) on the Snapshot | | A `json` entry with `moreTokens` and `keys` but no `body` | That entry, too large to send | Request `/s/?id=`, or open it in [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay) | | `meta.markdown` on a Highlights response | The full Main Markdown | Open its `url` | Everything else the capture saved stays one request away: `meta.resources` links to it. See [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). ## Next steps - [Errors & retries](https://expand.ai/docs/errors-and-retries): what each error means and whether to retry. - [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure): read `meta.resources` and the artifacts behind a Snapshot. - [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight): highlight a Fetch, choose what gets highlighted, and build citations. - [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay): inspect Snapshots and citations visually. - [API reference](https://expand.ai/api-reference): exact schemas and nested include defaults. --- # Overview (https://expand.ai/docs/fetch) Fetch renders live pages and returns agent-ready Markdown, State JSON, and replayable Snapshots with evidence. Send a browser page URL and get back context your agent can use: the page as clean Markdown, the data behind it as State JSON, and a saved Snapshot that every line links back to. Fetch also converts [files](https://expand.ai/docs/fetch/documents) and [YouTube videos](https://expand.ai/docs/fetch/youtube) to Markdown. **More answers, fewer tokens.** Other tools guess what matters from the HTML. Iris and X-Ray look at the rendered page, keep what your agent needs, and hold the rest one call away. ## When to use Fetch - **Agent research.** Your agent reads the page itself, not a summary, and every line traces back to the source. - **Pricing and product pages.** Tables and check marks come back as tables and check marks. - **JavaScript-heavy apps and dashboards.** Fetch renders the page in a real browser, and State JSON holds the data behind it. - **Docs ingestion.** Long pages come back whole, without navigation or footers. - **PDFs, files, and videos.** [PDFs, office files, ebooks, images, and audio](https://expand.ai/docs/fetch/documents) convert to Markdown on the same endpoint, with no flag to set. A [YouTube video](https://expand.ai/docs/fetch/youtube) comes back as its transcript. If your agent needs to click through a page rather than read it, browser automation is the better tool. See [Why Expand](https://expand.ai/docs/why-expand#when-expand-is-worth-it). ## Your first Fetch Give Fetch any URL, and it returns the page as Markdown your agent can actually reason over. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://news.ycombinator.com' }) ``` **CLI** ```bash expandai fetch https://news.ycombinator.com ``` **MCP** ```jsonc // Arguments for the fetch tool { "url": "https://news.ycombinator.com" } ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://news.ycombinator.com"}' ``` To get an API key and run this yourself, follow the [Quickstart](https://expand.ai/docs/quickstart). You can also call Fetch from the CLI, MCP, or your agent. See [Ways to use Expand](https://expand.ai/docs/quickstart#ways-to-use-expand). ## How Fetch works Most extraction tools make a bad trade. They keep everything and flood your agent's context, or keep only the main text and lose the details that matter. Fetch does neither. **Keep everything** Complete, but expensive. Raw payloads, navigation, footers, duplicated UI, and low-signal text crowd out the answer. **Keep only main text** Concise, but lossy. Tables, icons, app state, links, and source evidence disappear when the agent needs the next step. **Fetch** Useful first, complete when needed. Main Markdown and compact State JSON carry the common case. Progressive disclosure keeps the rest one call away. Rather than returning one flat stream, Fetch turns a page into layers. The first layer is short and answers most questions. Each subsequent layer goes deeper, so your agent reads only as much as the task requires. We call this [progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). ### Main Markdown The Main Markdown is the page's content, and the first layer your agent reads. [Iris](https://expand.ai/docs/fetch/iris), our vision model, sees the rendered page. It understands design and layout, so tables and icons stay intact instead of collapsing into loose paragraphs. Navigation and footers move to the [Appendix](https://expand.ai/docs/fetch#appendix). Ads and banners are dropped. ```markdown | Plan | Price | Seats | | --- | --- | --- | | Pro | $20 | 5 | | Team | $40 | 20 | ``` ### State JSON State JSON is the hidden data behind the page. It supplements the Main Markdown your agent reads. Prices, stock levels, and coordinates often live in API responses, not in the visible text. [X-Ray](https://expand.ai/docs/fetch/x-ray), our second model, finds that data and keeps what's useful. Analytics and configuration are dropped. ```json { "product": "PRS516", "price": 1850, "inStock": true } ``` ### Appendix The Appendix is everything around the content: navigation, sidebars, related links, and footers. Iris moves it out of the Main Markdown so your agent reads less. Appendix content remains available: a marker shows where each piece was, and your agent opens the Appendix when it needs a link or a menu. ```markdown # Main document [appendix omitted: 14 links] # Appendix - Navigation - Footer links ``` ### Snapshot & Evidence A browser page Snapshot is the page exactly as your agent saw it. Replay it or share it, even after the site changes. Snapshots live in the [Wayback Machine](https://expand.ai/docs/wayback-machine). Evidence links every line of the Main Markdown and State JSON back to its spot in the Snapshot, so you can check where each answer came from. A captured price such as "$1,850 list price" links to its source block in the saved Snapshot. Open the citation to see the evidence in replay. ## Everything runs on Fetch Fetch is powered by two System 1 models. Every other product is built on Fetch. - [Iris](https://expand.ai/docs/fetch/iris) writes the Markdown, and [X-Ray](https://expand.ai/docs/fetch/x-ray) finds the data behind the page. - [Wayback Machine](https://expand.ai/docs/wayback-machine) saves browser page Fetches as Snapshots you can replay and cite. - [Highlights](https://expand.ai/docs/highlights) returns ranked, cited passages that match your query. - [Crawl](https://expand.ai/docs/crawl) runs Fetch across a site as a stream of events. - [WebSync](https://expand.ai/docs/websync) keeps checking the pages in a crawl and streams their changes. ## Next steps - [Quickstart](https://expand.ai/docs/quickstart): run one Fetch. - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url): the Markdown text and the JSON object, and how to turn each part on or off. - [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay): inspect evidence visually. - [API reference](https://expand.ai/api-reference): use exact endpoint schemas. --- # Iris (https://expand.ai/docs/fetch/iris) The vision model for browser page Fetches. Iris looks at the rendered page and decides what goes in the Main Markdown, what moves to the Appendix, and what is dropped. Iris writes the [Main Markdown](https://expand.ai/docs/fetch#main-markdown) for pages Fetch opens in a browser. It looks at the rendered page, not at the raw HTML, and decides what your agent should read first. There is nothing to switch on for a browser page. [Files](https://expand.ai/docs/fetch/documents) and [YouTube videos](https://expand.ai/docs/fetch/youtube) use a different conversion path. ## Why a model Other tools turn a page into Markdown with hand-written rules that guess what matters from the HTML. The guesses fail in both directions. A rule that drops a footer removes a company's address on one page, and on the next page it does not fire and ships thousands of tokens of clutter. Iris does not guess from the HTML alone. It looks at the page the way you do. ## What Iris sees For every element on the rendered page, Iris sees where it sits, how large it is, how bold its text is, whether it is pinned or floating, whether something covers it, and whether it works as navigation or as content. So a pricing table stays a table, a check mark that means "included" stays a check mark, and a sidebar that looks like navigation is treated as navigation. ## What Iris does Iris sorts every element on the page into one of three groups. Its rule of thumb is that if a reasonable question could be asked about an element, the element stays. | Group | Definition | Where it ends up | | -------- | -------------------------------------------- | ----------------------------------------------------------------------- | | Keep | The content | The Main Markdown | | Appendix | Navigation, sidebars, footers, related links | The [Appendix](https://expand.ai/docs/fetch#appendix), one request away | | Drop | Ads, cookie banners, promos, share buttons | Nowhere. Dropped. | Iris cannot know what your agent will need, so it picks a good default and keeps the rest reachable. ## Where Iris shows up | Output | Markdown text | JSON object | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | The Main Markdown | The document body | `markdown` | | Where a piece of the Appendix was cut | A `[appendix omitted: 14 links · ~320 tokens] {612}` marker in the body. `{612}` is the evidence ID of the first omitted block. | The same marker in `markdown` | | A pointer to the Appendix | `appendix:` in the frontmatter, with its `url` and link counts | `meta.appendix` | | The Appendix itself, with `?include=appendix` | A `## Appendix` section at the end | A `## Appendix` section at the end of `markdown` | | An evidence ID for citations | The `{612}` in each marker | `json[].id` on State JSON entries, `location.evidenceId` on Highlights snippets | Every line of the Main Markdown links back to the exact spot on the captured page, so you can check what your agent read. See [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay#citation-links). How to include the Appendix on each surface is on [Include options](https://expand.ai/docs/fetch/fetching-a-url#include-options). ## Next steps - [How Fetch works](https://expand.ai/docs/fetch#how-fetch-works): where the Main Markdown sits among the other layers. - [X-Ray](https://expand.ai/docs/fetch/x-ray): the second model, for the data behind the page. - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url): the two modes, and what to include. - [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure): read the Appendix and everything else a Fetch held back. --- # X-Ray (https://expand.ai/docs/fetch/x-ray) The model for browser page Fetches that finds the data behind the page and returns what is useful as State JSON, next to the Main Markdown. X-Ray finds the data behind pages Fetch opens in a browser. It returns the page's embedded JSON and the JSON its own APIs loaded as [State JSON](https://expand.ai/docs/fetch#state-json), next to the [Main Markdown](https://expand.ai/docs/fetch#main-markdown), so your agent gets the data behind the page from the same call. There is nothing to switch on for a browser page. State JSON is on by default. [Files](https://expand.ai/docs/fetch/documents) and [YouTube videos](https://expand.ai/docs/fetch/youtube) have no State JSON. ## Why a model Not everything a page shows is in its HTML. Prices, stock levels, coordinates, and gate numbers often live in JSON the page loads. A Markdown converter never sees that JSON. The usual workaround is driving a full browser for every page, which is slow and expensive. ## What X-Ray sees - **In the page.** JSON-LD, framework state such as Next.js and Nuxt data, Apollo and other initial state, JSON script tags, and data attributes. - **On the network.** The JSON responses the page loaded from its own APIs. Each State JSON entry says which. `kind` is `inline` or `network`, and `role` names the source, such as `ld+json`, `next-data`, or `network-response`. ## What X-Ray does X-Ray reads the JSON a page carries and keeps the parts a reasonable question could be asked about. It drops analytics, configuration, and state the page only needs to draw itself. ## What X-Ray returns Each State JSON entry carries an evidence `id` that citations point to, its `kind` and `role`, the request `url` and `method` for network data, and the data itself in `body`. ```json { "id": 3, "kind": "network", "role": "network-response", "url": "https://www.figma.com/api/plans", "method": "GET", "body": { "plans": [{ "name": "Pro", "seats": 10 }] } } ``` When an entry is too large to send, it comes back without `body`, with `moreTokens` and a sample of its top-level `keys` instead. How to get the rest is on [References to more](https://expand.ai/docs/fetch/fetching-a-url#references-to-more). ## Where X-Ray shows up | Surface | Where it is | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | JSON object | `json`, one entry per piece of evidence. `meta.stateJson.moreTokens` says how much was held back. | | Markdown text | A `## Expand State JSON` section at the end of the document, one fenced `json` block per entry. `stateJson.moreTokens` in the frontmatter says how much was held back. | | URL | `?include=statejson` turns State JSON on, and an empty `?include=` turns it off. It is on by default. | | Request body | `include: { json: false }` turns State JSON off when the URL has no `?include=`. | | Batched Fetch | `data.json` on each successful page result. | How to turn State JSON off on each surface is on [Include options](https://expand.ai/docs/fetch/fetching-a-url#include-options). ## Example **TypeScript SDK** ```ts import { ExpandClient, isFetchOriginError } from '@expandai/sdk' const expand = new ExpandClient() const page = await expand.fetchJson({ url: 'https://www.figma.com/pricing/' }) if (!isFetchOriginError(page)) { console.log(page.json) // State JSON } ``` **CLI** ```bash expandai fetch https://www.figma.com/pricing/ --format json # json in the output is the State JSON ``` **MCP** ```jsonc // Arguments for the fetch tool { "url": "https://www.figma.com/pricing/" } // json in the result is the State JSON ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch/json \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://www.figma.com/pricing/"}' ``` ## Next steps - [How Fetch works](https://expand.ai/docs/fetch#how-fetch-works): where State JSON sits among the other layers. - [Iris](https://expand.ai/docs/fetch/iris): the first model, for the Main Markdown. - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url): the State JSON shape, its evidence fields, and when to ask for it. - [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay#urls): inspect State JSON next to the captured page. --- # YouTube videos (https://expand.ai/docs/fetch/youtube) Fetch a YouTube video and get its transcript back as Markdown, with title, description, chapters, and timestamps. Agents get sent to YouTube too. A conference talk, a tutorial, a product demo. The watch page itself is a player, a sidebar of recommendations, and a comment thread. Your agent needs what was said. Fetch recognizes YouTube video URLs automatically. Send the video URL and get the transcript back as Markdown, with title, channel, description, and chapters, instead of rendering the watch page. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://www.youtube.com/watch?v=aBcDeFgHiJk' }) ``` **CLI** ```bash expandai fetch https://www.youtube.com/watch?v=aBcDeFgHiJk ``` **MCP** ```jsonc // Arguments for the fetch tool { "url": "https://www.youtube.com/watch?v=aBcDeFgHiJk" } ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://www.youtube.com/watch?v=aBcDeFgHiJk"}' ``` ## Which URLs are recognized Any YouTube URL associated with one video is converted to transcript Markdown. Everything else is fetched as an ordinary page. `youtube.com`, `www.youtube.com`, `m.youtube.com`, `music.youtube.com`, and `youtube-nocookie.com` are all acceptable. Share noise in the query string (`si`, `t`, `feature`, a playlist `list` next to a `v`) is ignored. **The video ID is what matters.** Playlists, channels, `@handle` pages, and search results carry no single video ID, so they render in a browser and come back as a page. See [Browser-rendered pages](https://expand.ai/docs/fetch/browser-rendered-pages). | Form | Transcript | Example | | -------------- | ---------- | ------------------------------------------------------ | | Watch page | Yes | `https://www.youtube.com/watch?v=aBcDeFgHiJk` | | Short link | Yes | `https://youtu.be/aBcDeFgHiJk` | | Shorts | Yes | `https://www.youtube.com/shorts/aBcDeFgHiJk` | | Live replay | Yes | `https://www.youtube.com/live/aBcDeFgHiJk` | | Embed | Yes | `https://www.youtube.com/embed/aBcDeFgHiJk` | | Legacy player | Yes | `https://www.youtube.com/v/aBcDeFgHiJk` | | Playlist | No | `https://www.youtube.com/playlist?list=PLaBcDeFgHiJk` | | Channel | No | `https://www.youtube.com/@exampleconf` | | Search results | No | `https://www.youtube.com/results?search_query=keynote` | ## What comes back A video comes back in either of Fetch's two modes, as Markdown text or as a JSON object. See [Two modes](https://expand.ai/docs/fetch/fetching-a-url#two-modes). ### Markdown Mode Markdown Mode returns the transcript as Markdown text. The frontmatter has the same keys as any other page fetched as Markdown text, so no additional parsers are required. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://www.youtube.com/watch?v=aBcDeFgHiJk' }) ``` **CLI** ```bash expandai fetch https://www.youtube.com/watch?v=aBcDeFgHiJk ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://www.youtube.com/watch?v=aBcDeFgHiJk"}' ``` ```markdown --- url: "https://www.youtube.com/watch?v=aBcDeFgHiJk" capturedAt: "2026-09-08T10:15:00.000Z" pageType: "video" title: "Keynote: What changed in the browser this year" author: "Example Conf" createdAt: "2026-05-14" image: "https://i.ytimg.com/vi/aBcDeFgHiJk/maxresdefault.jpg" --- # Keynote: What changed in the browser this year Example Conf · 41:07 · Published 2026-05-14 · 128,304 views Transcript: captions (manual, en) ## Description Opening keynote from Example Conf 2026. Slides: https://example.com/slides ## Transcript ### [0:00] Welcome [0:04] Good morning, everyone. Thanks for being here so early on a Thursday. [0:21] I want to start with a number. Last year, 38 percent of the pages we measured shipped a table the crawler never saw. ### [2:15] The rendering gap [2:15] Here is what that looks like in practice. ... ``` Reading it top to bottom: - **Summary line.** Channel, length, publish date, and view count. Anything YouTube doesn't report is left out. - **`Transcript:` line.** Where the text came from, such as `captions (manual, en)`. See [Transcription waterfall](https://expand.ai/docs/fetch/youtube#transcription-waterfall). - **Description.** The uploader's description, with links. Left out when there is none. - **Chapters.** `### [m:ss] Title` headings, only when the uploader set chapters. - **Paragraphs.** Each starts with its timestamp, such as `[0:42]` or `[1:02:03]`. Sound-only captions like `[Music]` are dropped. ### JSON Mode JSON Mode returns the transcript as a JSON object. It has all the standard fields plus `data.video`, which holds the video's details as fields, so you don't have to pull them out of the Markdown. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const page = await expand.fetchJson({ url: 'https://www.youtube.com/watch?v=aBcDeFgHiJk' }) ``` **CLI** ```bash expandai fetch https://www.youtube.com/watch?v=aBcDeFgHiJk --format json ``` **MCP** ```jsonc // Arguments for the fetch tool { "url": "https://www.youtube.com/watch?v=aBcDeFgHiJk" } ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch/json \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://www.youtube.com/watch?v=aBcDeFgHiJk"}' ``` ```json { "meta": { "version": 1, "url": "https://www.youtube.com/watch?v=aBcDeFgHiJk", "capturedAt": "2026-09-08T10:15:00.000Z", "snapshotId": "0f9c3a1e-...", "contentType": "text/html" }, "markdown": "---\nurl: \"https://www.youtube.com/watch?v=aBcDeFgHiJk\"\n...", "json": [], "data": { "video": { "provider": "youtube", "videoId": "aBcDeFgHiJk", "title": "Keynote: What changed in the browser this year", "channel": "Example Conf", "channelUrl": "https://www.youtube.com/@exampleconf", "durationSeconds": 2467, "publishedAt": "2026-05-14", "viewCount": 128304, "language": "en", "transcript": { "source": "captions", "kind": "manual", "language": "en" } }, "document": { "url": "https://s3.amazonaws.com/.../document/original.json?X-Amz-Signature=...", "contentType": "application/json", "sizeBytes": 48211 }, "usage": { "documentPages": 0, "ocrPages": 0, "audioSeconds": 0, "videos": 1 } } } ``` | Field | Definition | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `data.video.videoId` | The eleven-character ID, whichever URL form you sent. | | `data.video.title`, `channel`, `channelUrl` | As YouTube reports them. `channel` and `channelUrl` are absent when unknown. | | `data.video.durationSeconds` | Length of the video, not of the transcript. | | `data.video.publishedAt` | Upload date as an ISO calendar date. | | `data.video.viewCount` | View count at capture time. | | `data.video.language` | The video's primary spoken language, when YouTube or the transcript reports one. | | `data.video.transcript` | `source`, `kind`, and `language` of the transcript. See [Transcription waterfall](https://expand.ai/docs/fetch/youtube#transcription-waterfall). | | `data.video.storyboard` | Preview frames sampled through the video, as sprite sheet images. Absent when YouTube doesn't publish them. | | `data.document.url` | A presigned link to what the transcript was made from: the caption track as JSON, or the audio that was transcribed. | | `data.usage.audioSeconds` | Seconds of audio transcribed. `0` on the captions path. | | `meta.url` | Always the canonical `watch?v=` URL. | | `meta.contentType` | `text/html`, because that is what YouTube served. The transcript is Expand's rendering of it, the same way a PDF's Markdown is. | **Note:** A video isn't rendered in a browser, so it has no State JSON, Appendix, links, screenshot, or replay. Asking for them doesn't cause an error. `json` comes back as an empty list, and the other fields are left out. To watch the video while you read the transcript, open the Fetch in the [Playground](https://expand.ai/docs/wayback-machine/playground-and-replay). It shows the transcript next to an embedded player of the video. ## Transcription waterfall Fetch uses existing captions first and transcribes the audio only when there are none. It tries, in order: 1. The creator's captions, in the video's native language. 2. YouTube's auto-generated captions, in the video's native language. 3. The creator's captions, in English. 4. YouTube's auto-generated captions, in English. 5. Any other captions the creator uploaded. 6. If none of those exist, Fetch downloads the audio and transcribes it with a speech model. Both modes say where the transcript came from: the `Transcript:` line in Markdown text, and `data.video.transcript` in the JSON object. For a German video, the first two steps return `source: "captions"`, `language: "de"`, and `kind: "manual"` or `"auto"`. English captions use `language: "en"`; another uploaded track uses its own language. If Fetch transcribes the audio, it returns `source: "transcription"` with no `kind`, and `data.usage.audioSeconds` records the transcribed duration instead of `0`. | Field | Values | Definition | | ---------- | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `source` | `captions` or `transcription` | Whether Fetch used a caption track from YouTube or transcribed the audio with a speech model. | | `kind` | `manual` or `auto` | When `source` is `captions`, says whether the track is the creator's own (`manual`) or YouTube's speech recognition (`auto`). | | `language` | A BCP-47 language tag (`en`, `de`, `pt-BR`) | The transcript's language, when known. | **Note:** You can't request a specific language yet. Fetch prefers the video's own language over a translation, so a German video with German captions comes back in German. ## Additional credit usage Fetching a YouTube video costs `6` credits (`$0.006`), whatever its length. That's 1 credit for the Fetch request plus 5 credits for the video, which cover the transcript, timestamps, chapters, and preview frames. When a video has no caption track, Fetch transcribes the audio with a speech model, which charges an additional `10` credits per minute: | Unit | Credits | When it applies | | ----- | --------------- | ------------------------------------------------------------------ | | Video | `5` per video | Every delivered transcript. One per Fetch, whatever its length. | | Audio | `10` per minute | Only when no caption track exists. Rounded up, one-minute minimum. | For example, a 25-minute video with captions costs `6` credits. The same video with no captions costs `256` credits. You are only charged for a delivered result, never when a video can't be fetched, such as a private video or one longer than `maxAudioSeconds`. See [Video errors](https://expand.ai/docs/fetch/youtube#video-errors). `data.usage` reports `videos` and `audioSeconds`, so you can check how each Fetch was charged. See [Pricing & Usage](https://expand.ai/docs/pricing-and-usage#what-you-are-charged-for-by-outcome). ## Limiting long transcriptions When a video has no captions, Fetch transcribes the audio with a speech model. That's the only cost that grows with the video's length. You can cap it with the optional `documentConfig.maxAudioSeconds` field. | Field | Definition | Type | Default | Range | | ----------------- | ----------------------------------------------------------------- | ------- | ------- | -------------- | | `maxAudioSeconds` | Refuse the Fetch rather than transcribe a video longer than this. | integer | `3600` | `1` to `14400` | **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const markdown = await expand.fetch({ url: 'https://youtu.be/aBcDeFgHiJk', documentConfig: { maxAudioSeconds: 900 }, }) ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://youtu.be/aBcDeFgHiJk", "documentConfig": { "maxAudioSeconds": 900 } }' ``` - When captions are available, Fetch ignores `maxAudioSeconds`. A video of any length comes back in full for the flat `6` credits, even a six-hour livestream replay. - When captions aren't available, Fetch checks the video's length against `maxAudioSeconds` before it downloads any audio. If the video is longer than the limit, the request is refused and you aren't charged. **Recommendation:** Raise the limit only as far as the videos you expect. The ceiling is four hours (`14400` seconds). A transcription that is within the limit can still take a while. **Note:** Transcription can run for up to 55 minutes once it starts. Only the wait for a worker uses the [150-second budget](https://expand.ai/docs/fetch/browser-rendered-pages#readiness-and-timing). Set your HTTP client timeout to allow a long transcription to finish. ## Batched Fetch [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch) doesn't return YouTube transcripts yet. It renders the watch page in a browser and returns it as a standard page, with no transcript. Send video URLs to `POST /v1/fetch` or `POST /v1/fetch/json` until this lands. ## Video errors On top of the [standard Fetch errors](https://expand.ai/docs/errors-and-retries), a video Fetch can return three more errors. Their full payloads are in the [API reference](https://expand.ai/api-reference/errors#document-and-video-errors-404-410-413-415-422-502-503). | `_tag` | Code | Retryable | | --------------------------------------------------------------------------------------------------------------- | ----- | --------- | | [`FetchVideoUnavailable`](https://expand.ai/docs/fetch/youtube#fetchvideounavailable) | `422` | Sometimes | | [`FetchDocumentProcessingUnavailable`](https://expand.ai/docs/fetch/youtube#fetchdocumentprocessingunavailable) | `503` | Yes | | [`FetchPageTooLarge`](https://expand.ai/docs/fetch/youtube#fetchpagetoolarge) | `413` | No | ### `FetchVideoUnavailable` **HTTP 422 · Retryable: sometimes** YouTube won't serve this video, so there's no transcript to return. ```json { "_tag": "FetchVideoUnavailable", "url": "https://www.youtube.com/watch?v=aBcDeFgHiJk", "availability": "live", "message": "This live event will begin in 2 hours" } ``` The `availability` field explains why the video is unavailable, and whether it's retryable: | Value | Definition | Retryable | | --------------- | ---------------------------------------------------------- | --------- | | `private` | Private. | No | | `removed` | Deleted, or its account was terminated. | No | | `geoRestricted` | Not available from Expand's region. | No | | `ageRestricted` | Behind an age gate that needs a signed-in account. | No | | `membersOnly` | Reserved for channel members or Premium subscribers. | No | | `live` | Streaming right now. There is no transcript until it ends. | Yes | | `upcoming` | A scheduled premiere or stream that has not started. | Yes | | `unknown` | Withheld for a reason YouTube did not name. | No | This error is not retryable, except when `availability` is `live` or `upcoming`. Fetch those again after the stream ends, once YouTube has processed the replay. ### `FetchDocumentProcessingUnavailable` **HTTP 503 · Retryable: yes** YouTube refused the request, or the speech model couldn't finish. ```json { "_tag": "FetchDocumentProcessingUnavailable", "url": "https://www.youtube.com/watch?v=aBcDeFgHiJk", "operation": "video", "message": "YouTube refused the request" } ``` The `operation` field is set to `video` when YouTube refused the request, such as a bot check or a rate limit, and `transcription` when the speech model couldn't finish. The TypeScript SDK retries this error automatically, with backoff. ### `FetchPageTooLarge` **HTTP 413 · Retryable: no** The video has no captions and is longer than `maxAudioSeconds`. ```json { "_tag": "FetchPageTooLarge", "url": "https://www.youtube.com/watch?v=aBcDeFgHiJk", "limitKind": "audio_seconds", "limit": 3600, "measured": 5400 } ``` For a video, the `limitKind` field is set to `audio_seconds`, and `limit` and `measured` are in seconds. To fetch the video, increase the `maxAudioSeconds` limit, up to the ceiling of four hours (`14400` seconds). A video longer than that can't be transcribed. See [Limiting long transcriptions](https://expand.ai/docs/fetch/youtube#limiting-long-transcriptions). ## Next steps - [PDFs and other files](https://expand.ai/docs/fetch/documents): PDFs, images, office files, ebooks, and audio on the same endpoint. - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url#include-options): choose what a Fetch returns. - [Pricing & Usage](https://expand.ai/docs/pricing-and-usage): see what transcribed minutes cost in credits. - [Errors & retries](https://expand.ai/docs/errors-and-retries): which errors to retry, and what you are charged for. --- # Highlights (https://expand.ai/docs/highlights) Narrow a Fetch to what matches your query and get ranked, cited snippets instead of the whole document. Even accurate Markdown can run to thousands of tokens. Your agent costs more, responds more slowly, and answers less accurately. Highlights takes a query and returns only the passages that answer it, ranked and cited, from the [Main Markdown](https://expand.ai/docs/fetch#main-markdown), [State JSON](https://expand.ai/docs/fetch#state-json), and, optionally, the [Appendix](https://expand.ai/docs/fetch#appendix). On a fictional pricing page, the complete page would take 3,200 tokens. Highlights returns only matching snippets: | Query | Ranked snippets (source, score, evidence ID) | Snippet tokens | | ---------------- | ------------------------------------------------------------------------------------------------------------------ | -------------- | | `pro plan seats` | Pro pricing row (Main Markdown, 0.94, 1204); `plans.pro.seats: 10` (State JSON, 0.91, 3) | 18 | | `refund policy` | "Cancel within 30 days for a full refund." (Main Markdown, 0.88, 1311) | 11 | | `free trial` | "Every plan starts with a 14-day trial." (Main Markdown, 0.83, 1290); Free pricing row (Main Markdown, 0.61, 1203) | 22 | The Pro pricing snippet keeps the table shape: ```md | Plan | Price | Seats | | --- | --- | --- | | Pro | $20 | 10 seats | ``` The rest of the page is held back. These values illustrate the result shape and token difference; they are not a benchmark or Expand's pricing. ## Where Highlights shows up Fetch Highlights runs while a page is fetched. Snapshot Highlights runs on a Snapshot you already have. For each, this table shows what to call on every surface and where the snippets come back: | | Fetch Highlights | Snapshot Highlights | | ----------------------- | -------------------------------------------------- | -------------------------------------------- | | API | `/v1/fetch` or `/v1/fetch/json`, with `highlights` | `POST /v1/snapshots//highlights` | | TypeScript SDK | `fetchJson()` or `fetch()`, with `highlights` | `snapshots.get()`, with `highlights` | | CLI | `expandai fetch --highlights` | `expandai snapshots get --highlights` | | MCP | `fetch` tool, with `highlights` | `get_snapshot` tool, with `highlights` | | Results are returned in | `data.highlights` | `highlights` | Fetch Highlights is a normal Fetch with a Highlights query added. Snapshot Highlights adds the same `highlights` option to a Snapshot read and takes a `snapshotId` instead of a URL. Without `highlights`, a Snapshot read returns its public catalog. ## Next steps - [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight): Fetch Highlights and Snapshot Highlights, with code, and how to tune the results. - [Working with snippets](https://expand.ai/docs/highlights/snippets): what a snippet contains, how to cite it, and how to read beyond it. - [Wayback Machine](https://expand.ai/docs/wayback-machine): the Snapshots Highlights searches. --- # Working with snippets (https://expand.ai/docs/highlights/snippets) A snippet is one passage of the page that matches your query. Highlights returns a ranked list of snippets, under highlights.snippets, instead of the whole document. A snippet can come from the Main Markdown, State JSON, or Appendix. A snippet is one passage that matched your query, with its source, its score, and a link back to where it sits on the page. Here, a query for `plan limit` matches a row in the [Main Markdown](https://expand.ai/docs/fetch#main-markdown) and the value behind it in the [State JSON](https://expand.ai/docs/fetch#state-json): ```json { "highlights": { "query": "plan limit", "snippets": [ { "source": "markdown", "text": "| Plan | Projects | Seats |\n| --- | --- | --- |\n| Pro | 10 projects | 50 seats |", "score": 0.94, "location": { "evidenceId": 1204 } }, { "source": "statejson", "text": "planLimits.pro.projects: 10", "json": { "plan": "pro", "projects": 10, "seats": 50 }, "score": 0.91, "location": { "evidenceId": 3, "jsonPath": "$.planLimits.pro" } } ] } } ``` | Field | Definition | | --------------------- | --------------------------------------------------------------------------------------- | | `text` | The passage itself, readable. | | `source` | Where it came from: `markdown`, `statejson`, or `appendix`. | | `score` | How well it matches the query, from 0 to 1. Snippets are ranked by it. | | `json` | For a State JSON snippet, the structured value behind the match, when it fits inline. | | `location.evidenceId` | The exact spot on the page it came from, when available. It turns into a citation link. | | `location.jsonPath` | For a State JSON snippet, where the value sits in the State JSON, when available. | **Note:** No kind of snippet, and no snippet order, is guaranteed. Highlights returns only what scores above `minScore`, and ranking can change as the capture, the layers, and the ranker improve. ## Citing a snippet Every snippet's `location.evidenceId` becomes a link that opens the exact spot on the page: ```txt https://expand.land/s/?id= ``` With the API, build the link from `location.evidenceId`. The CLI and MCP add it for you, as `citationUrl`. How to inspect a citation is on [Playground & replay: Citation links](https://expand.ai/docs/wayback-machine/playground-and-replay#citation-links). ## Reading beyond a snippet When a snippet does not provide enough context, you can read the snippet's surrounding content or read the entire Main Markdown. Neither creates a new Fetch nor costs any credits. No API key is needed. ```bash # Read around one snippet, using its evidence ID. curl -H "Accept: text/markdown" "https://expand.land/s//content?id=" # Read the full Main Markdown. curl -H "Accept: text/markdown" "https://expand.land/s//content" ``` To see everything inside a Snapshot, such as the original HTML and the network requests, see [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). ## Next steps - [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight): get snippets as you fetch, or from a Snapshot you already have. - [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay): open a citation and see the snippet in place. - [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure): everything else a Snapshot saves. --- # Two ways to highlight (https://expand.ai/docs/highlights/two-ways-to-highlight) Highlights can be used on new Fetch requests, as well as on existing Snapshots. We call them Fetch Highlights and Snapshot Highlights, respectively. Highlights runs in two places. Fetch Highlights runs as part of a Fetch, and Snapshot Highlights runs later, on a page you fetched before. ## Fetch Highlights Add a query to a Fetch to get matching [snippets](https://expand.ai/docs/highlights/snippets) from the whole page. **Note:** Fetch Highlights reads the layers your Fetch includes: the [Main Markdown](https://expand.ai/docs/fetch#main-markdown) and [State JSON](https://expand.ai/docs/fetch#state-json) by default. The [Appendix](https://expand.ai/docs/fetch#appendix) is highlighted only if you include it. To choose the layers, see [Fetching a URL: Include options](https://expand.ai/docs/fetch/fetching-a-url#include-options). The MCP `fetch` tool always includes Main Markdown when searching a new page. It can turn State JSON off or the Appendix on, but it has no `includeMarkdown` argument. The MCP `get_snapshot` tool can choose all three layers when searching an existing Snapshot. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const page = await expand.fetchJson({ url: 'https://docs.stripe.com/webhooks', highlights: { query: 'signature verification', maxResults: 5 }, }) ``` **CLI** ```bash expandai fetch https://docs.stripe.com/webhooks --highlights "signature verification" --max-results 5 ``` **MCP** ```jsonc // Arguments for the fetch tool { "url": "https://docs.stripe.com/webhooks", "highlights": { "query": "signature verification", "maxResults": 5 } } ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch/json \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://docs.stripe.com/webhooks", "highlights": { "query": "signature verification", "maxResults": 5 } }' ``` **JSON object** ```json { "meta": { "url": "https://docs.stripe.com/webhooks", "snapshotId": "019...", "playground": "https://expand.land/s/019...", "markdown": { "url": "https://expand.land/s/019.../content" } }, "markdown": "", "json": [], "data": { "highlights": { "query": "signature verification", "snippets": [ { "source": "markdown", "text": "Verify the event's signature with the endpoint's signing secret before you process it.", "score": 0.93, "location": { "evidenceId": 512 } }, { "source": "statejson", "text": "webhooks.signing.header: Stripe-Signature", "json": { "header": "Stripe-Signature", "tolerance": 300 }, "score": 0.87, "location": { "evidenceId": 7, "jsonPath": "$.webhooks.signing" } } ] } } } ``` **Markdown text** ````markdown --- url: "https://docs.stripe.com/webhooks" query: "signature verification" more: "https://api.expand.ai/v1/public/snapshots/01JPUBLIC01?format=markdown" --- # Highlights ## 1. Markdown Verify the event's signature with the endpoint's signing secret before you process it. ## 2. State JSON: $.webhooks.signing ```json { "header": "Stripe-Signature", "tolerance": 300 } ``` ```` **Note:** In the JSON object, snippets come back under `data.highlights`, and the usual layers are left out: the full Markdown, State JSON, Appendix, and screenshot. `meta.markdown.url` links to the full Markdown if you need it. In Markdown text, the response is the snippets themselves. ## Snapshot Highlights Add `highlights` to a Snapshot read to highlight an existing [Snapshot](https://expand.ai/docs/fetch#snapshot--evidence) with a new question, using the `meta.snapshotId` from a prior Fetch response. It doesn't create a new Fetch, so it costs only the 1 Highlights credit. A Fetch response carries [two IDs](https://expand.ai/docs/wayback-machine#two-ids), and Snapshot Highlights takes `meta.snapshotId`, the capture's own ID. Snapshot Highlights always answers with a JSON object, and [Batched Fetch](https://expand.ai/docs/fetch/batched-fetch) doesn't support it. The `include` options select the layers to highlight: - `include.markdown` highlights the Main Markdown. - `include.json` highlights the State JSON. - `include.appendix` highlights the Appendix when Appendix text exists. - When `include` isn't provided, Highlights uses the Main Markdown and State JSON. The Appendix is opt-in. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const result = await expand.snapshots.get({ snapshotId: '', highlights: { query: 'retry policy', maxResults: 5 } }) ``` **CLI** ```bash # The CLI always highlights Main Markdown and State JSON. # Use the SDK, API, or MCP to choose other layers. expandai snapshots get --highlights "retry policy" --max-results 5 ``` **MCP** ```jsonc // Arguments for the get_snapshot tool { "snapshotId": "", "highlights": { "query": "retry policy", "maxResults": 5 } } ``` **cURL** ```bash curl "https://api.expand.ai/v1/snapshots//highlights" \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "highlights": { "query": "retry policy", "maxResults": 5 } }' ``` By default this highlights Main Markdown and State JSON. Add `include: { markdown: true, json: true, appendix: true }` to the SDK or API request to highlight all three. When an `include` object is present, list every layer you want: omitted layers are off. MCP uses `includeMarkdown`, `includeJson`, and `includeAppendix` booleans instead; its defaults are `true`, `true`, and `false`. **Illustrative response**. Snapshot Highlights puts the snippets in `highlights`, while Fetch Highlights uses `data.highlights`. ```json { "snapshotId": "019...", "response": { "url": "https://docs.stripe.com/webhooks", "originStatusCode": 200 }, "highlights": { "query": "retry policy", "snippets": [ { "source": "markdown", "text": "Stripe retries failed deliveries for up to three days, with an exponential backoff.", "score": 0.9, "location": { "evidenceId": 640 } }, { "source": "statejson", "text": "webhooks.retries.window: 3 days", "json": { "window": "3 days", "backoff": "exponential" }, "score": 0.84, "location": { "evidenceId": 9, "jsonPath": "$.webhooks.retries" } } ] }, "durationMs": 107 } ``` **Warning:** A request with nothing to highlight is rejected with `400 BadRequest`. ## Tuning the results When you highlight, you can control the `minScore`, the minimum relevance score, and the `maxResults`, the most snippets you can receive that score above the `minScore`. `minScore` defaults to `0.6` (or `0` in MCP) and can be set from `0` to `1`. `maxResults` defaults to `5` and can be set from `1` to `50`. **Note:** Zero snippets is not an error. The layers were valid, but nothing scored above `minScore`. Lower `minScore`, or include more layers. ### For Fetch Highlights Set how many snippets come back, and how relevant they must be, in the same request that fetches the page. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const result = await expand.fetchJson({ url: 'https://docs.stripe.com/webhooks', highlights: { query: 'signature verification', maxResults: 10, minScore: 0.75 } }) ``` **CLI** ```bash # The CLI always highlights Main Markdown and State JSON. # Use the SDK, API, or MCP to choose other layers. expandai fetch https://docs.stripe.com/webhooks --highlights "signature verification" --max-results 10 --min-score 0.75 ``` **MCP** ```jsonc // Arguments for the fetch tool { "url": "https://docs.stripe.com/webhooks", "highlights": { "query": "signature verification", "maxResults": 10, "minScore": 0.75 } } ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch/json \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://docs.stripe.com/webhooks", "highlights": { "query": "signature verification", "maxResults": 10, "minScore": 0.75 } }' ``` ### For Snapshot Highlights Set the same limits when you highlight a Snapshot you already have, to get only the matching snippets you want from it. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const result = await expand.snapshots.get({ snapshotId: '', highlights: { query: 'retry policy', maxResults: 10, minScore: 0.75 } }) ``` **CLI** ```bash # The CLI always highlights Main Markdown and State JSON. # Use the SDK, API, or MCP to choose other layers. expandai snapshots get --highlights "retry policy" --max-results 10 --min-score 0.75 ``` **MCP** ```jsonc // Arguments for the get_snapshot tool { "snapshotId": "", "highlights": { "query": "retry policy", "maxResults": 10, "minScore": 0.75 } } ``` **cURL** ```bash curl "https://api.expand.ai/v1/snapshots//highlights" \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "highlights": { "query": "retry policy", "maxResults": 10, "minScore": 0.75 } }' ``` ## Highlights errors On top of the [standard Fetch errors](https://expand.ai/docs/errors-and-retries), Highlights can return two more errors. Their full payloads are in the [API reference](https://expand.ai/api-reference/errors). | `_tag` | Code | Retryable | | ---------------------------------------------------------------------------------------------------------------------------- | ----- | --------- | | [`BadRequest`](https://expand.ai/docs/highlights/two-ways-to-highlight#badrequest) | `400` | No | | [`FetchSnapshotHighlightsNotFound`](https://expand.ai/docs/highlights/two-ways-to-highlight#fetchsnapshothighlightsnotfound) | `404` | No | ### `BadRequest` **HTTP 400 · Retryable: no** The Highlights request has nothing to highlight. ```json { "_tag": "BadRequest", "message": "Highlights requires at least one included corpus: markdown, json, or appendix." } ``` Turn at least one layer back on. See [Snapshot Highlights](https://expand.ai/docs/highlights/two-ways-to-highlight#snapshot-highlights). ### `FetchSnapshotHighlightsNotFound` **HTTP 404 · Retryable: no** No Snapshot with that `snapshotId` exists for your organization. ```json { "_tag": "FetchSnapshotHighlightsNotFound", "snapshotId": "019..." } ``` **Recommendation:** Check the `snapshotId`. It is `meta.snapshotId` from the Fetch, not the public `meta.resources.snapshotId`. ## Next steps - [Working with snippets](https://expand.ai/docs/highlights/snippets): what a snippet contains, how to cite it, and how to read beyond it. - [Fetching a URL: Include options](https://expand.ai/docs/fetch/fetching-a-url#include-options): choose which layers a Fetch includes, and so which Fetch Highlights searches. - [Errors & retries](https://expand.ai/docs/errors-and-retries): what to do when a request fails. --- # Wayback Machine (https://expand.ai/docs/wayback-machine) Browser page Fetches save a Snapshot you can replay, cite, and read again without fetching twice. Each browser page Fetch saves a Snapshot: the rendered page and its assets, with the [Main Markdown](https://expand.ai/docs/fetch#main-markdown), [State JSON](https://expand.ai/docs/fetch#state-json), and evidence made from it. When the live site changes, the Snapshot still shows what your agent read. A successful Fetch response carries a `snapshotId`. With a browser page Snapshot you can: - **Replay** the captured page at `expand.land/s/`, or open it next to the Markdown in the Playground. - **Cite** it. [Highlights](https://expand.ai/docs/highlights) and the [Playground](https://expand.ai/docs/wayback-machine/playground-and-replay) point to the exact source block or State JSON entry behind an answer. - **Read it again** without fetching twice. Highlights and [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure) work on the Snapshot you already have. - **Download** the whole capture as a WACZ archive. **Note:** A [PDF or other file](https://expand.ai/docs/fetch/documents) and a [YouTube video](https://expand.ai/docs/fetch/youtube) get a `snapshotId` too, but no replayable page or WACZ archive. ## What a Snapshot saves For a browser page, the first response carries the Main Markdown and State JSON. Its Snapshot also keeps everything else from the capture, and each part is one public `GET` away, with no API key: | Part | What it is for | | ----------------- | ----------------------------------------------------------------------------------- | | The original HTML | Re-parse it with your own tools, or diff it against a later Fetch. | | A HAR | Every HTTP exchange behind the page, to debug a bad capture or replay it elsewhere. | | Network records | The JSON APIs the page loaded, so you can call them directly next time. | | Assets | The images, PDFs, and scripts the page referenced. | | Evidence | The source blocks and State JSON entries that citations point to. | | A WACZ archive | The whole capture, which powers replay. | How to read each part is on [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). ## Two IDs A browser page Fetch response carries two different IDs for its Snapshot: | ID | Where | Used by | | --------------------------- | ---------------------------------------------------------------- | ---------------------------------------------- | | `meta.snapshotId` | The JSON object, and `snapshotId` in Markdown text frontmatter | Highlights, the Playground, and replay | | `meta.resources.snapshotId` | The JSON object, and the `more` URL in Markdown text frontmatter | The public read URLs on Progressive disclosure | **Warning:** They are not interchangeable. Highlights with the public ID returns `404`, and the public read URLs do not accept the capture ID. ## Who can read a Snapshot Snapshot links are shareable citation links. Treat them like public links to captured source material. - Snapshot URLs are public. Anyone who has a link can open it. - Snapshot pages ask search engines not to index them, and send no referrer. That is not access control, and it may not apply to non-HTML responses such as `/content` or `archive.wacz`. **Recommendation:** Don't fetch private pages you don't want shared as public links. Keep secrets, tokens, and private customer data out of pages you plan to cite. ## Getting the replay link A browser page Fetch's JSON object carries the replay link as `meta.playground`. In Markdown text, it is the `playground` line of the frontmatter. **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const page = await expand.fetchJson({ url: 'https://news.ycombinator.com' }) console.log(page.meta.playground) // https://expand.land/s/ng0EFjewTRF6 ``` **CLI** ```bash expandai fetch https://news.ycombinator.com --format json # meta.playground in the output is the replay link ``` **MCP** ```jsonc // Arguments for the fetch tool { "url": "https://news.ycombinator.com" } // meta.playground in the result is the replay link ``` **cURL** ```bash curl https://api.expand.ai/v1/fetch/json \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://news.ycombinator.com"}' ``` ## Next steps - [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay): replay a Snapshot, inspect it next to its Markdown, and cite the evidence in it. - [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure): read everything a Snapshot saves, from code or from an agent. - [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight): highlight a Snapshot you already have, with `meta.snapshotId`. --- # Playground & replay (https://expand.ai/docs/wayback-machine/playground-and-replay) Replay the page Fetch captured, see where each line of Markdown came from, and give your users a link to the exact evidence behind an answer. A Snapshot can be opened by a person. Replay shows the page as Fetch captured it. The Playground puts that replay next to the Markdown, so you can see where each line came from. A citation link opens one piece of evidence, so a reader can check an agent's claim against the page. ## The Playground layout The Playground is two synchronized panes: the Markdown on the left, the replayed page on the right. The left pane opens in raw Markdown; you can switch to rendered Markdown or copy the Markdown. Hovering or selecting a source block in the Markdown highlights the region it came from in the replay. A [State JSON](https://expand.ai/docs/fetch#state-json) citation opens the matched State JSON entry instead. ## URLs Every URL is built from the `snapshotId`: | What you get | URL | | ------------------------ | ---------------------------------------------------- | | Replay of the whole page | `https://expand.land/s/` | | The Playground | `https://expand.land/s//playground` | | One piece of evidence | `https://expand.land/s/?id=` | | The WACZ archive | `https://expand.land/s//archive.wacz` | **Replay** opens the captured page from the Snapshot's WACZ archive, so you see the page state Fetch saw. **Evidence** is one block of the page. Add `?id=` to the replay URL and it opens there: a Markdown or [Appendix](https://expand.ai/docs/fetch#appendix) block is highlighted in the replay, and a State JSON entry opens as JSON. **Note:** The Playground may change the URL's hash while it synchronizes its panes. That hash is not a citation link. Citation links always use `?id=`. ## Citation links A citation link is more than a footnote. It lets a reader open the exact captured evidence behind an agent's claim. Every [Highlights](https://expand.ai/docs/highlights) snippet carries a `location.evidenceId`, whatever its source. The citation link is the replay URL plus that ID: ```txt https://expand.land/s/?id= ``` How each surface gives you the link is on [Working with snippets: Citing a snippet](https://expand.ai/docs/highlights/snippets#citing-a-snippet). You do not need a snippet to cite. Expand stamps an evidence ID into the captured Markdown next to the text it belongs to, written as `{1199}`, and every response carries the `playground` URL for the Snapshot. Any citation is those two joined. That works for anything you read, not only snippets, which is why the markers are inline in the Markdown. If there is no precise evidence, link to the whole Snapshot at `/s/`. ## Reading a Snapshot as text The same URLs serve machines. A browser gets the replay, and a client that asks for Markdown gets the [Main Markdown](https://expand.ai/docs/fetch#main-markdown): ```bash # The full Main Markdown. curl -H "Accept: text/markdown" "https://expand.land/s//content" # One block, by its evidence ID. A State JSON entry comes back as JSON. curl -H "Accept: text/markdown" "https://expand.land/s//content?id=" ``` An invalid or missing evidence ID returns not found. For everything else a Snapshot saves, such as the original HTML and the network requests, see [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure). ## WACZ archive The WACZ archive is the whole capture: the page, its assets, the Markdown, the State JSON, and the evidence. It is what replay loads from, and you can download it from `/s//archive.wacz`. **Note:** The Snapshot service does not serve screenshots. Fetch returns a screenshot when you request one. See [Browser-rendered pages: Screenshots](https://expand.ai/docs/fetch/browser-rendered-pages#screenshots). ## Embedding evidence in your product If you build a research agent, return citation links with the answer, so your users can click a claim and inspect the source behind it. A citation card needs the claim, the link, a source label, and the snippet text: ```txt Enterprise includes 50 seats. [View evidence] Source: pricing page Evidence: State JSON Link: https://expand.land/s/019...?id=3 ``` To build it: - Store the `snapshotId` with the research result. - Store each snippet's `source`, `text`, `location`, and `citationUrl` when there is one. - With the API, build the link from `location.evidenceId`. The CLI and MCP give you `citationUrl`. - Show the links in your product's UI, not only in logs. **Note:** This is an example of a UI you could build on Expand. The Playground itself does not render citation cards. Snapshot links are public to anyone who has them. See [Who can read a Snapshot](https://expand.ai/docs/wayback-machine#who-can-read-a-snapshot) before you share them. ## Next steps - [Working with snippets](https://expand.ai/docs/highlights/snippets): what a snippet contains, and how to cite one. - [Progressive disclosure](https://expand.ai/docs/wayback-machine/progressive-disclosure): read everything a Snapshot saves, from code or from an agent. - [Browser-rendered pages](https://expand.ai/docs/fetch/browser-rendered-pages): how a page is captured in the first place. --- # Progressive disclosure (https://expand.ai/docs/wayback-machine/progressive-disclosure) Fetch once, then go back to the Snapshot for more. Read the original HTML, the network requests, or the whole archive without fetching the page again. Fetch gives your agent the useful part first: the [Main Markdown](https://expand.ai/docs/fetch#main-markdown) and [State JSON](https://expand.ai/docs/fetch#state-json). The Snapshot keeps everything else from the capture, and each part is one public `GET` away, with no API key. That is progressive disclosure: start small, and read more only when you need it. The trail runs from the answer to the bytes. The Fetch response names the Snapshot's catalog, `meta.resources`. The catalog lists collections, such as the network requests, and artifacts, such as the original HTML. A collection holds records, an artifact offers a bounded preview, and both end in the saved content. The catalog serves two different jobs: - **Code.** You fetched pages earlier and now want more from those Snapshots: the HAR to debug, the network records to find the API behind a page, the original HTML to re-parse. - **Agents.** The Markdown answered most of the question, but one value is missing. The agent looks it up in the saved HTML instead of fetching the page again. ## Where the trail starts Every Fetch response names the Snapshot's catalog. Markdown text carries a `more` URL in its frontmatter. The JSON object carries the same URL in `meta.resources`. Follow it exactly as given. **JSON object** ```json { "meta": { "resources": { "version": 1, "snapshotId": "01JPUBLIC01", "url": "https://api.expand.ai/v1/public/snapshots/01JPUBLIC01", "markdownUrl": "https://api.expand.ai/v1/public/snapshots/01JPUBLIC01?format=markdown" } } } ``` **Markdown text** ```md --- snapshotId: 019c25ee-1a2a-7d6d-9c1b-3f4e5a6b7c8d # the capture ID, for Highlights and the Playground more: https://api.expand.ai/v1/public/snapshots/ng0EFjewTRF6?format=markdown # the public ID is in the URL --- ``` **Warning:** Every URL on this page uses `meta.resources.snapshotId`, the public ID. It is not the `meta.snapshotId` that Highlights and the Playground use. See [Two IDs](https://expand.ai/docs/wayback-machine#two-ids). ## Reading a Snapshot from code Store `meta.resources.snapshotId` next to each Fetch result. Hours or days later, everything the capture saved is still readable by that ID, with no API key and no new browser session. | You need | Read | | --------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | The HTTP exchanges behind the page, to debug a bad capture or replay it elsewhere | `artifacts/har/content` | | The JSON API a page loads, so you can call it directly next time | `collections.network` for the request, or the HAR for every exchange | | The original HTML, to re-parse with your own tools or diff against a later Fetch | `artifacts/html.original/content` | | The files the page referenced: images, PDFs, scripts | `collections.assets` | | The whole capture, as one archive | the `wacz` artifact | The catalog lists what was saved and where to read it: **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const catalog = await expand.snapshots.get({ snapshotId: '' }) ``` **cURL** ```bash curl 'https://api.expand.ai/v1/public/snapshots/' ``` ```json { "version": 1, "snapshotId": "01JPUBLIC01", "collections": { "evidence": "/v1/public/snapshots/01JPUBLIC01/evidence", "network": "/v1/public/snapshots/01JPUBLIC01/network", "assets": "/v1/public/snapshots/01JPUBLIC01/assets", "artifacts": "/v1/public/snapshots/01JPUBLIC01/artifacts" }, "artifacts": [ { "id": "html.original", "kind": "html", "url": ".../artifacts/html.original/content", "preview": ".../artifacts/html.original/preview" }, { "id": "har", "kind": "har", "url": ".../artifacts/har/content", "preview": ".../artifacts/har/preview" }, { "id": "wacz", "kind": "wacz", "url": ".../wacz" } ], "links": { "markdown": "/v1/public/snapshots/01JPUBLIC01?format=markdown", "json": "/v1/public/snapshots/01JPUBLIC01" } } ``` The links are paths on the API origin. The catalog has two kinds of entry: - **Collections** are lists, returned as `{ items, next? }`. A network or asset record carries the captured request and response headers, status, and timing, plus an `artifact` link to the HAR and a `content` link to the saved body. A record's `url` says where the bytes came from. It is never a download target. - **Artifacts** are saved bytes. `preview` gives a bounded, base64-encoded look, and `url` gives the full content. **Note:** Requests to the same URL share one `content` link, which returns the first match. For polling or repeated POSTs, read the HAR, which keeps every exchange. A preview answers `413` when the archive is over the server's budget. The `content` link still works, so fall through to the full download. With the TypeScript SDK, one `fetchJson` runs the browser and everything after reads from the Snapshot: ```ts import { ExpandClient, isFetchOriginError } from '@expandai/sdk' const expand = new ExpandClient() // 1. The answer: metadata and Markdown. The only step that runs a browser. const page = await expand.fetchJson({ url: 'https://example.com' }) if (isFetchOriginError(page) || !page.meta.resources) throw new Error('No Snapshot to read') // 2. The catalog behind meta.resources: what else was saved. No API key needed. const snapshotId = page.meta.resources.snapshotId const catalog = await expand.snapshots.get({ snapshotId }) console.log(catalog.artifacts.map((artifact) => artifact.id)) // ['html.original', 'har', 'wacz'] // 3. A collection: every network request the page made. const network = await expand.snapshots.network({ snapshotId }) // 4. A bounded look at an artifact before downloading all of it. const preview = await expand.snapshots .preview({ snapshotId, artifactId: 'html.original', maxBytes: 512 }) .catch(() => undefined) // 5. The full bytes. Catalog links are paths on the API origin. const har = catalog.artifacts.find((artifact) => artifact.id === 'har') if (har) { const bytes = await fetch(new URL(har.url, page.meta.resources.url)).then((response) => response.arrayBuffer()) } await expand.close() ``` ## Letting an agent verify against the page Main Markdown leaves out the clutter on purpose. Sometimes the thing the task needs went with it: an attribute value, a `data-` field, a number inside a script tag, a hidden element, a form default. The agent should not fetch the page again. The saved HTML is exact, already paid for, and one `GET` away. ```bash # 1. The Markdown response names the catalog. No API key for anything below. MORE=$(sed -n 's/^more: //p' page.md | tr -d '"') curl -sS "$MORE" # 2. Look at the first bytes of the original HTML before reading all of it. curl -sS "https://api.expand.ai/v1/public/snapshots//artifacts/html.original/preview?maxBytes=4096" \ | jq -r .body | base64 -d # 3. Read the full source and search it. curl -sS "https://api.expand.ai/v1/public/snapshots//artifacts/html.original/content" \ | grep -o 'data-price="[^"]*"' ``` If the value was loaded by an XHR rather than rendered into the HTML, list `collections.network` to find the request, then read its `content` link for the saved body. **Note:** This is a different tool from Highlights. Highlights finds passages that are in the Markdown and State JSON but were too long to read. The HTML is not in that corpus. When the Markdown does not contain the value at all, go to `html.original`. Rules for this path: - `GET` the `more` URL exactly as written, with no auth header. Follow its links, and never guess routes. - Preview before content. Read `truncated` and fall through to `content` when the preview is cut off or answers `413`. - Treat a record's `url` as where the bytes came from. Saved bytes come from the catalog's artifact links and a record's `content` link, never from the original URL. - Never fetch the origin again to fill a gap. Cite the Snapshot the value came from. **Recommendation:** Install the Expand Skill with [`expandai skill install`](https://expand.ai/docs/cli-skill-and-hooks). It already teaches agents the `more` catalog step. ## Next steps - [Fetching a URL](https://expand.ai/docs/fetch/fetching-a-url): the fields in the first response. - [Two ways to highlight](https://expand.ai/docs/highlights/two-ways-to-highlight): highlight the Markdown and State JSON of a Snapshot you already have. - [Playground & replay](https://expand.ai/docs/wayback-machine/playground-and-replay): open the same Snapshot as a person. - [TypeScript SDK](https://expand.ai/docs/typescript-sdk#progressive-disclosure): the `snapshots` methods. --- # WebSync (https://expand.ai/docs/websync) A crawl that keeps going. Expand rechecks every page on the schedule you set and streams each change as an event. WebSync is a [crawl](https://expand.ai/docs/crawl) that keeps going. Add `sync` to a crawl and Expand rechecks every page on the schedule you set, and streams each change as an event, in order, with an offset you can resume from. Use it to keep a site in sync with your database or your file system. ## Starting a synced crawl **TypeScript SDK** ```ts import { ExpandClient } from '@expandai/sdk' const expand = new ExpandClient() const [crawl] = await expand.crawls.create( { sources: ['https://example.com/pricing'], sync: { freshness: '24h' } }, { idempotencyKey: 'pricing-watch' }, ) for await (const event of crawl.events({ from: savedOffset, live: true })) { // Apply the event to your copy, then persist event.offset. } ``` **CLI** ```bash # Keeps a folder of Markdown files current. The SDK and API stream events instead. expandai sync https://example.com/pricing ./pricing --every 24h ``` **cURL** ```bash curl https://api.expand.ai/v1/crawls \ -H "x-expand-api-key: $EXPAND_API_KEY" \ -H "x-idempotency-key: pricing-watch" \ -H "Content-Type: application/json" \ -d '{"sources":["https://example.com/pricing"],"sync":{"freshness":"24h"}}' # Wait for new events after the offset you saved. curl "https://api.expand.ai/v1/crawls/$CRAWL_ID/events?offset=$SAVED_OFFSET&live=long-poll" \ -H "x-expand-api-key: $EXPAND_API_KEY" ``` The `sync` settings: | Setting | Definition | Default | | ---------------------- | -------------------------------------------------------------------------------------------------------------- | -------- | | `freshness` | How often every page is checked for changes, such as `24h` or `7d`. How each check is made is up to the crawl. | Required | | `renderEvery` | How often a page is rendered in full, even when a check saw no change. | `7d` | | `budget.rendersPerDay` | The most pages rendered in a day. | None | A synced crawl keeps its stream open. `crawl.events()` returns once you have caught up, and `live: true` keeps waiting for new events. Everything else about a crawl, such as how it finds pages, its states, and its errors, is on [Crawl](https://expand.ai/docs/crawl). ## What counts as a change Some parts of a page always change: a rotating banner, a timestamp, a view count. `changePolicy` tells WebSync which parts to leave out of the comparison, so they do not trigger a change: - `changePolicy.ignore` removes the matching parts of the page before comparing. - `changePolicy.volatile` replaces them with stable markers. Both take CSS selectors, and both leave the saved page content as it was. ## When a page changes A `page.changed` event means a page differs from its last capture. **TypeScript SDK** ```ts for await (const event of crawl.events({ from: savedOffset, live: true })) { if (event.type !== 'page.changed') continue console.log(event.url, event.changed) // { content: true, stateJson: false, extraction: false } console.log(event.diff.markdown) } ``` **cURL** ```bash curl "https://api.expand.ai/v1/crawls/$CRAWL_ID/events?offset=$SAVED_OFFSET&live=long-poll" \ -H "x-expand-api-key: $EXPAND_API_KEY" ``` | Field | Definition | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `changed` | Which parts changed: `content`, `stateJson`, or `extraction`. | | `diff.stateJson` | Each changed State JSON path with its old and new value, as `{ from, to }`. A price that went from 49 to 59 shows up here as one path. | | `diff.markdown` | A diff of the Markdown. | | `diff.truncated` | `true` when the diff was too large to send in full. | | `captureId`, `previousCaptureId` | The new capture and the one before it. Read either in full with `expand.captures(id)`. | ## When a page disappears A `page.removed` event means a page is gone. WebSync does not decide this from one bad check. It sends the event only after at least three checks agree. | Field | Definition | | ----------------------------- | ------------------------------------------------------------------------------------------------------- | | `evidence.kind` | Why: `http-404`, `http-410`, `content-gone`, or `redirected-away`. | | `evidence.checks` | The checks behind the decision, each with its `captureId`, `observedAt`, HTTP `status`, and `finalUrl`. | | `evidence.sitemapDisappeared` | `true` when the page also dropped out of the site's sitemap. | | `previousCaptureId` | The last capture of the page while it existed. | ## A page's history Every capture of a page is kept as a version. Read them newest first: **TypeScript SDK** ```ts for await (const version of expand.pages(event.pageId).versions()) { console.log(version.observedAt, version.http.status, version.contentSha256) } ``` **cURL** ```bash curl https://api.expand.ai/v1/pages/$PAGE_ID/versions \ -H "x-expand-api-key: $EXPAND_API_KEY" ``` Each version has its `captureId`, `observedAt`, the stream `offset` it was published at, a `contentSha256` of its content, the HTTP `status` and `finalUrl`, a `quality` of `complete`, `best-effort`, or `degraded`, and the page `meta`. Pass `at` with a timestamp to get the version that was current at that moment. To compare any two captures of a page, not only neighbours: **TypeScript SDK** ```ts const diff = await expand.pages(pageId).diff({ from: olderCaptureId, to: newerCaptureId }) ``` **cURL** ```bash curl "https://api.expand.ai/v1/pages/$PAGE_ID/diff?from=$OLDER_CAPTURE_ID&to=$NEWER_CAPTURE_ID" \ -H "x-expand-api-key: $EXPAND_API_KEY" ``` It returns the same `diff` as a `page.changed` event. ## Credit usage A check is free. Every page Expand renders costs `1` credit, the same as a Fetch. `budget.rendersPerDay` caps what a synced crawl can spend in a day. See [Pricing & Usage](https://expand.ai/docs/pricing-and-usage). ## Next steps - [Crawl](https://expand.ai/docs/crawl): how a crawl finds pages, its states, and its errors. - [TypeScript SDK: Crawl and WebSync](https://expand.ai/docs/typescript-sdk#crawl-and-websync): every option, method, and event field. - [Pricing & Usage](https://expand.ai/docs/pricing-and-usage): what a synced crawl costs, and how credits are billed. - [Errors & retries](https://expand.ai/docs/errors-and-retries): the standard errors every request can return.