Expand AI logo
DocsDocs
Glow Active
API Reference
Login

Documentation

Get started

IntroductionWhy ExpandQuickstartAuthenticationConnect your agent (MCP)

Products

OverviewFetching a URLBatched Fetch
API reference

SDKs & tools

TypeScript SDK
MCP tools & resourcesDocs for agents

Reference

API referenceErrors & retriesPricing & UsageLimitsConcurrency tiersFAQ
Browse docs

Get started

IntroductionWhy ExpandQuickstartAuthenticationConnect your agent (MCP)

Products

OverviewFetching a URLBatched Fetch
API reference

SDKs & tools

TypeScript SDK
MCP tools & resourcesDocs for agents

Reference

API referenceErrors & retriesPricing & UsageLimitsConcurrency tiersFAQ

TypeScript SDK

Call Fetch, Highlights, Batched Fetch, Crawl, and WebSync from TypeScript with @expandai/sdk, and turn the results into citation links.

@expandai/sdk is a typed client for the Expand API. It reads your API key from the environment, retries transient failures, and returns the same Markdown text and JSON object the API does.

npm install @expandai/sdk
pnpm add @expandai/sdk
bun add @expandai/sdk
import { ExpandClient } from '@expandai/sdk'

const
expand = new ExpandClient()
const markdown = await expand.fetch({ url: 'https://news.ycombinator.com' })

Install and auth

Set EXPAND_API_KEY and the client picks it up. Pass apiKey only when the key comes from somewhere else, such as a secrets manager. The constructor throws when it finds no key, so a missing key fails at startup rather than on the first request.

export EXPAND_API_KEY="xpnd_..."
import { ExpandClient } from '@expandai/sdk'

const expand = new ExpandClient({
  apiKey: process.env.EXPAND_API_KEY,
  timeoutMs: 180_000,
})
OptionDefinitionDefault
apiKeyYour API key, sent as the x-expand-api-key header.process.env.EXPAND_API_KEY
baseUrlWhere requests go.https://api.expand.ai
Note
fetch(), fetchJson() with Fetch parameters, snapshots.get(), and the Batched Fetch methods take timeoutMs and maxRetries in their final options argument to override the client's defaults for one call. snapshots.get() uses the same timeout and retry options. Other Crawl and Snapshot methods have their own options.

To create a key, see Authentication.

Method index

MethodDefinitionReturns
fetch(params, options?)Markdown Mode. One URL in, Markdown text out.string
fetchJson(params, options?)JSON Mode. Main Markdown, State JSON, the Snapshot, and Fetch Highlights.

Fetch

JSON Mode with fetchJson

fetchJson() calls /v1/fetch/json and returns a JSON object: the Main Markdown in markdown, the State JSON in json, and the Snapshot in meta. Use it when your code reads the fields separately.

When the origin never answered, the result is a FetchOriginError instead of content. Narrow with isFetchOriginError() before reading the fields.

import { ExpandClient, isFetchOriginError } from '@expandai/sdk'

const expand = new ExpandClient()

const page = await expand.fetchJson({ url: 'https://www.figma.com/pricing/' })

if (isFetchOriginError(page)) {
  console.error(page.kind, page.retryable)
} else {
  console.log(page.markdown)
  console.log(page.json)
  console.log(page.meta.playground)
}
FieldDefinition
markdownThe Main Markdown.
jsonThe State JSON, one entry per piece of evidence.
meta.snapshotIdThe capture ID. Pass it to snapshots.get.
Note

isFetchOriginError() only catches an origin that never answered. Its kind names the failure: dns, tls, connectionRefused, connectionReset, connectionClosed, addressUnreachable, emptyResponse, or connectionTimeout. An origin that answered with an error still returns content, and reports the status in originError.

The full shape is in the API reference.

Markdown Mode with fetch

fetch() calls /v1/fetch and returns Markdown text as one string, with the metadata in YAML frontmatter at the top. Use it when the result goes straight into a prompt.

const markdown = await expand.fetch({ url: 'https://www.figma.com/pricing/' })

Include options

Both methods take an include string in their second argument. It is the URL's ?include= parameter, and it turns State JSON and the Appendix on and off. The request body's include does the same with appendix and json when you leave the string out, and in JSON Mode it also asks for extras outside the document, such as a screenshot.

const withAppendix = await expand.fetch(
  { url: 'https://www.figma.com/pricing/' },
  { include: 'appendix,statejson' },
)

const withScreenshot = await expand.fetchJson(
  { url: 'https://www.figma.com/pricing/', include: { screenshot: { fullPage: true } } },
  { include: 'appendix' },
)
Note

{ include: null } leaves the parameter off the URL, the same as not passing it. What each token does, and the defaults, are on Fetching a URL.

Batched Fetch

batched() starts a run and returns its id. getBatched() polls it. Each result carries its own status next to its data, so a URL that failed or was cancelled does not hide the ones that worked.

const run = await expand.batched({
  urls: ['https://www.figma.com/pricing/', 'https://www.figma.com/about/'],
})

let status = await expand.getBatched(run.id, { limit: '10', offset: '0' })

while (status.batchedStatus === 'QUEUED' || status.batchedStatus === 'RUNNING') {
  await new Promise((resolve) => setTimeout(resolve, 1000))
  status =
  • getBatched() takes limit, offset, and include as strings, because they are URL parameters.
  • When you leave out idempotencyKey, the SDK generates one and reuses it for its own retries. Pass a stable key to make separate calls, or calls after a restart, land on the same run.
  • Reusing a key with a different payload returns 409 BatchedIdempotencyConflict.

The run's lifecycle, its options, and the result shape are on Batched Fetch.

Highlights

Fetch Highlights

Add highlights to a JSON Mode Fetch and the snippets come back under data.highlights.snippets, next to the whole page.

const page = await expand.fetchJson({
  url: 'https://docs.stripe.com/webhooks',
  highlights: { query: 'signature verification', maxResults: 5, minScore: 0.6 },
})

if (!isFetchOriginError(page)) {
  for (const snippet of page.data?.highlights?.snippets ?? []) {
    console.log(snippet.source, snippet.score, snippet.text)
  }
}

Each snippet has source (markdown, appendix, or statejson), text, score, and location. A State JSON snippet also has json, the matched value. The SDK does not add a citationUrl to snippets. Build one with the citation helpers.

Read a Snapshot

const catalog = await expand.snapshots.get({ snapshotId: '<snapshotId>' })
console.log(catalog.links.markdown)

This reads the public catalog without sending credentials. Adding highlights switches to an authenticated POST and costs one Highlights credit. The return type follows the options you pass.

Snapshot Highlights

snapshots.get() with highlights calls /v1/snapshots/<snapshotId>/highlights and highlights a Snapshot you already have, without fetching the page again. Pass meta.snapshotId from the earlier Fetch.

const page = await expand.fetchJson({ url: 'https://docs.stripe.com/webhooks' })

if (!isFetchOriginError(page)) {
  const result = await expand.snapshots.get({
    snapshotId: page.meta.snapshotId,
    highlights: { query: 'retry policy', maxResults: 5, minScore: 0.6 },
    include: { markdown: true, json: true, appendix: false },
  })

  console.log(result.highlights.snippets)
}

The result has highlights.snippets in the same shape as above, plus snapshotId, playground, response, and durationMs.

Warning

Without include, Highlights uses the Main Markdown and State JSON. When you pass include, a layer you leave out is off. Turning all three off returns 400 BadRequest.

How the two ways differ, and how to tune maxResults and minScore, is on Two ways to highlight.

Wayback Machine

Citation helpers

A snippet's location points at the exact spot on the Snapshot. The helpers turn it into a link a person can open.

import { citationUrl, resolvePlaygroundHost } from '@expandai/sdk'

if (!isFetchOriginError(page)) {
  const host = resolvePlaygroundHost(page.meta.playground, 'https://expand.land')

  for (const snippet of page.data?.highlights?.snippets ?? []) {
    console.log(snippet.text, citationUrl(page.meta.snapshotId, snippet.location, host))
  }
}
HelperDefinition
citationUrl(snapshotId, location, host?)The citation link for one snippet, from location.evidenceId.
playgroundBase(snapshotId, host?)The Playground link for the whole Snapshot.
playgroundContentUrl(snapshotId, evidenceId?, host?)The Main Markdown as text, or one piece of it.

What each link shows is on Playground & replay.

Progressive disclosure

A Fetch with a saved Snapshot returns meta.resources. Its snapshotId is the public ID, and the snapshots methods read the catalog behind it without an API key and without fetching the page again.

const page = await expand.fetchJson({ url: 'https://www.figma.com/pricing/' })
if (isFetchOriginError(page) || !page.meta.resources) throw new Error('No Snapshot was saved.')

const snapshotId = page.meta.resources.snapshotId
const catalog = await expand.snapshots.get({ snapshotId })
const network = await expand.snapshots.network({ snapshotId })
const preview = await expand.snapshots.preview({ snapshotId, artifactId: 'html.original', maxBytes: 512 })
Note

Catalog links are paths on the API origin, so resolve them against meta.resources.url before downloading. snapshots.network() and snapshots.preview() return unknown, so narrow the value before using it.

What the catalog holds and how an agent reads it is on Progressive disclosure.

Crawl and WebSync

crawls.create() starts a crawl. It returns one crawl per registrable domain in your sources, and needs an idempotency key you keep. crawls(id) picks up a crawl you started earlier, and crawls.merge() reads several as one stream.

const crawls = await expand.crawls.create(
  { sources: ['https://example.com/products'], formats: ['markdown', 'stateJson'] },
  { idempotencyKey: 'catalog-import-2026-09' },
)

for await (const event of expand.crawls.merge(crawls).events()) {
  // Handle page.added, page.changed, page.removed, page.failed, then persist event.offset.
}
MethodDefinition
crawl.events({ from?, live?, signal? })Stream page events. Pass a saved offset as from to resume. live: true keeps waiting on a synced crawl.
crawl.status(), crawl.done()Read the crawl's state, or wait until it is done, failed, or canceled.
crawl.update({ maxPages?, sync?, changePolicy? })Raise limits or change sync settings while the crawl is open.

Crawl errors are a CrawlClientApiError with code, status, and retryable. An offset too old to resume from throws CrawlOffsetExpiredError.

Crawl options

OptionDefinitionDefault
sources1 to 1,000 starting points. A URL string, or { url, kind } where kind is 'seed' (follow links from here) or 'sitemap' (read the URLs it lists).Required
formatsWhat each page event carries: any of markdown, , .

Crawl events

Every event has offset, id, observedAt, crawlId, and type.

TypeSent whenFields it adds
page.addedA new page was captured.pageId, url, captureId, discoveredBy (seed, sitemap, link, or search), depth, meta,
  • page.changed says what changed in changed (content, stateJson, extraction) and how in diff: changed State JSON paths with their old and new values, and a Markdown diff.
  • page.removed says why in evidence.kind: http-404, http-410, content-gone, or redirected-away, with the checks behind it.
  • page.failed gives a reason: blocked, not-found, timeout, error, robots-disallowed, robots-unreachable, or . It carries no content. Pages left out by never produce one.

event.page holds the formats you asked for when you created the crawl: markdown, stateJson, html.

Content in an event is capped at 64 KiB, filled in this order: State JSON, Markdown, the diff, then HTML. A field that does not fit is left out and named in event.page.truncated. The full capture is always available by its captureId:

const markdown = await expand.captures(event.captureId).markdown()
curl https://api.expand.ai/v1/captures/$CAPTURE_ID/markdown \
  -H "x-expand-api-key: $EXPAND_API_KEY"

event.meta carries the page's title, description, canonical URL, language, page type, and Open Graph data.

What a crawl is, how it finds pages, its states, and its errors are on Crawl. What a change looks like is on WebSync.

Errors and retries

ExpandClient throws one of four errors. Catch ExpandClientError to handle any of them.

ErrorDefinition
ExpandClientErrorThe base class. Also thrown for invalid client options.
ExpandClientApiErrorThe API answered with a 4xx or 5xx. Read status and body.
ExpandClientConnectionErrorThe API could not be reached.
import { ExpandClientApiError } from '@expandai/sdk'

try {
  await expand.fetchJson({ url: 'https://www.figma.com/pricing/' })
} catch (error) {
  if (error instanceof ExpandClientApiError) {
    console.error(error.status, error.body)
  }
  throw error
}

The SDK retries a failed attempt up to maxRetries times, with backoff, inside the call's timeoutMs:

  • Timeouts, connection errors, 408, 409, 429, and 5xx are retried. A 409 is retried unless it is a Batched Fetch conflict, BatchedIdempotencyConflict or BatchedCancellationConflict, which the SDK never retries.
  • The 502 page errors (FetchNavigationFailed, FetchCaptureFailed) and 504 FetchCaptureTimeout are not. All are verdicts about the page, and a retry would re-run a full capture for the same outcome. A 504 does not prove the page was at fault, so a deliberate manual retry can still make sense.
  • 529 FetchCapacityTimeout is retried, honoring its Retry-After: 5.

What each error means, and whether to retry it, is on Errors & retries.

Choosing timeoutMs

timeoutMs is one budget for the whole call, retries and backoff included, and the SDK enforces it on your side. A Fetch can take up to its 150-second budget plus a few seconds to answer, so the 60000 default cuts off long captures with ExpandClientTimeoutError before Expand answers.

Recommendation

Set timeoutMs to 180_000 for a single attempt. Add another 180 seconds per retry you allow, or set maxRetries: 0 when you only want one verdict.

const page = await expand.fetchJson(
  { url: 'https://www.figma.com/pricing/' },
  { timeoutMs: 180_000, maxRetries: 0 },
)

Effect

ExpandService is the same client for applications already built on Effect. It exposes the same methods, with failures typed in the error channel as ExpandSdkError, ExpandApiError, ExpandConnectionError, or ExpandTimeoutError.

import { NodeRuntime } from '@effect/platform-node'
import { Effect } from 'effect'
import { ExpandService, isFetchOriginError } from '@expandai/sdk'

const program = Effect.gen(function* () {
  const expand = yield* ExpandService
  const page = yield* expand.fetchJson({ url: 'https://www.figma.com/pricing/' })
  if (isFetchOriginError(page)) {
    yield* Effect.logWarning(page.kind, page.retryable)

ExpandService.layer() takes the same options as ExpandClient. You do not need it to use the SDK from plain Promise code.

Next steps

  • Fetching a URL: the two modes, what each layer holds, and the include options.
  • Two ways to highlight: Fetch Highlights and Snapshot Highlights, and how to tune them.
  • Errors & retries: what each error means and whether to retry.
  • API reference: the exact request and response schemas.
PreviousWebSync: Overview
NextCommands

On This Page

Install and authMethod indexFetchJSON Mode with fetchJsonMarkdown Mode with fetchInclude optionsBatched FetchHighlightsFetch HighlightsRead a SnapshotSnapshot HighlightsWayback MachineCitation helpersProgressive disclosureCrawl and WebSyncCrawl optionsCrawl eventsErrors and retriesChoosing timeoutMsEffectNext steps
timeoutMs
The time budget for one call, across every attempt. See Choosing timeoutMs.
60000
maxRetriesHow many times a failed attempt is retried.2
FetchJsonResponse
snapshots.get(params, options?)Read the public catalog, or add highlights for relevant passages from the Snapshot.SnapshotCatalog or SnapshotHighlightsResponse, inferred from highlights
batched(params, options?)Start a Batched Fetch.The run's id
getBatched(id, options?)Poll a run for its status and results.GetBatchedResponse
cancelBatched(id, options?)Stop a run.CancelBatchedResponse
snapshots.get, snapshots.network, snapshots.previewRead a Snapshot's public catalog, its network requests, or a slice of one artifact. No API key needed.See Progressive disclosure
crawls.create, crawls(id), crawls.mergeStart a crawl, pick one up by ID, or read several as one stream.CrawlResource
captures(id), pages(id)A capture's full content, or a page's version history.Capture and page readers
close()Abort active requests and release the client's connections.Promise<void>
meta.resources.snapshotId
The public Snapshot ID. Pass it to snapshots. See Two IDs.
meta.playgroundThe Playground link, for a human to open.
data.highlightsFetch Highlights, when the request had a highlights.
data.screenshotThe screenshot's url, when the request asked for one.
originErrorThe origin's HTTP error, when it answered with a status of 400 or above. The content is still there.
await
expand.getBatched(run.id, { limit:
'10'
, offset:
'0'
})
}
console.log(status.results)
playgroundAppendixUrl(snapshotId, host?)
The Appendix as text.
resolvePlaygroundHost(metaPlayground, fallback)The host from meta.playground, or fallback when it is missing.
parseSnapshotHandle(handle)The snapshotId and host from a Snapshot ID or a Playground link.
const
har = catalog.artifacts.find((artifact)
=>
artifact.id ===
'har'
)
if (har) {
const bytes = await fetch(new URL(har.url, page.meta.resources.url)).then((response) => response.arrayBuffer())
}
crawl.cancel()Stop the crawl.
crawl.snapshot()Read the current set of pages, then continue events from the offset it ends on.
crawl.pages(), crawl.page(id), crawl.pageByUrl(url)Browse the pages the crawl has kept.
expand.captures(id).markdown(), .html(), .stateJson()Read a capture's full content when an event's was cut at the 64 KiB cap.
expand.pages(id).versions(), .diff({ from, to })Read a synced page's history, or compare two captures.
stateJson
html
['markdown', 'stateJson']
maxPagesPages the crawl keeps, up to 1,000,000.25,000
maxDepth (per source)Clicks from the source URL, 0 to 100.50
include / exclude (per source)Path globs. include decides what is kept, and exclude keeps the crawl out.None
maxPages (per source)A cap for one source. Up to 32 sources may set their own.None
changePolicyCSS selectors to ignore when deciding whether a page changed. See WebSync.None
syncKeeps the crawl running on a schedule. See WebSync.None
page
page.changedA page changed since its last capture. Sent by WebSync.pageId, url, captureId, previousCaptureId, changed, diff, meta, page
page.removedA page is gone. Sent only after at least three checks agree.pageId, url, previousCaptureId, evidence
page.failedA page could not be captured.url, reason, optional detail, retriesExhausted
out-of-scope
include
ExpandClientTimeoutError
The call ran past timeoutMs.
} else {
yield* Effect.log(page.markdown)
}
})
NodeRuntime.runMain(program.pipe(Effect.provide(ExpandService.layer())))