Expand AI logo
DocsDocs
Glow Active
API Reference
Login

Documentation

Get started

IntroductionWhy ExpandQuickstartAuthenticationConnect your agent (MCP)

Products

OverviewFetching a URLBatched Fetch
API reference
Overview

SDKs & tools

TypeScript SDK
MCP tools & resourcesDocs for agents

Reference

API referenceErrors & retriesPricing & UsageLimitsConcurrency tiersFAQ
Browse docs

Get started

IntroductionWhy ExpandQuickstartAuthenticationConnect your agent (MCP)

Products

OverviewFetching a URLBatched Fetch
API reference
Overview

SDKs & tools

TypeScript SDK
MCP tools & resourcesDocs for agents

Reference

API referenceErrors & retriesPricing & UsageLimitsConcurrency tiersFAQ

Crawl

Give Crawl a URL and get the pages it finds within your limits, each one a full Fetch, as a stream you can resume.

Give Crawl a URL and it fetches the pages it finds within your limits. Each page it keeps is a full Fetch, with Main Markdown, State JSON, and a Snapshot. Pages arrive one at a time as events on a stream, so you can process them as they come and pick up where you left off.

A crawl stays on the URL's registrable domain and its subdomains. A crawl starting at docs.example.com can also find pages on shop.example.com.

How a crawl finds pages

  • maxDepth counts clicks from your URL. 0 is the URL alone. 1 adds linked pages and pages found through the site's sitemap or search, which count as one click. The default, 50, allows discovery up to .
maxPages
  • A kind: 'sitemap' source is the exception. The URLs it lists start at depth 0, as if each were your URL.
  • include decides what ends up in the results, not where the crawl goes. The crawl may open a page that does not match so it can follow its links. It does not keep that page, and the page does not count toward maxPages.
  • exclude keeps the crawl out. A matching page is never opened, so nothing behind it is found through it.

    • Starting a crawl

    import { ExpandClient } from '@expandai/sdk'

const expand = new ExpandClient()

const [crawl] = await expand.crawls.create(
  { sources: ['https://example.com/products'], formats: ['markdown', 'stateJson'] },
  { idempotencyKey: 'catalog-import-2026-09' },
)

for await (const event of crawl.events()) {
  if (event.type !==

    Every request needs an idempotency key. Sending the same key with the same request returns the crawls it already created instead of starting new ones. Sending it with a different request is rejected with 409 idempotency_key_conflict.

    Recommendation
    Persist the idempotency key, so a retry or a restart lands on the crawls you already started.

    create returns a list. Sources are grouped by domain, and each domain becomes its own crawl, so two sites in one request come back as two crawls. expand.crawls.merge(crawls) reads their events as one stream.

    Warning
    Every source must resolve to a public address. Otherwise the request is rejected with 422 crawl_blocked.

    Every option, with its default and limit, is on TypeScript SDK: Crawl options.

    What you get back

    A crawl does not return one big result. It streams events as it finds pages, in order. There are four kinds:

    EventSent when
    page.addedA new page was captured. It carries the page in the formats you asked for.
    page.changedA page changed since its last capture. Sent by WebSync.
    page.removedA page is gone.

    Every event carries an offset. Store it after you have handled the event, and pass it as from to pick up where you left off:

    for await (const event of crawl.events({ from: savedOffset })) {
  // ...
}
    Note
    When you resume, you may see an event you already handled. Use event.id to skip duplicates. If an offset is too old to resume from, the stream answers 410 crawl_stream_offset_expired. Start again from the beginning, or read the current set of pages with crawl.snapshot() and continue from the offset it ends on.

    The content in an event is capped at 64 KiB. A field that does not fit is left out and named in event.page.truncated. The full capture is always available by its captureId:

    const markdown = await expand.captures(event.captureId).markdown()
    curl https://api.expand.ai/v1/captures/$CAPTURE_ID/markdown \
  -H "x-expand-api-key: $EXPAND_API_KEY"

    The fields each event carries are on TypeScript SDK: Crawl events.

    Crawl states

    Read a crawl's state with crawl.status(), or wait for it to finish with crawl.done(). Over HTTP, it is GET /v1/crawls/:id.

    StateDefinition
    crawlingFinding and capturing pages.
    doneFinished. Nothing is left to find or capture.
    syncingFinished its first pass and is rechecking pages on its WebSync schedule.

    A crawl without sync closes for good when it is done, fails, or is canceled. Create a new crawl to run it again.

    Why a crawl stops early

    A finished crawl reports whether it covered everything it found. When it did not, coverage lists the causes:

    CauseWhat happenedWhat to do
    max-pagesMore pages were found than maxPages allows.Raise maxPages while the crawl is still open.
    source-limitA source hit its own maxPages.Raise that source's cap.

    Changing or stopping a crawl

    While a crawl is open, you can raise maxPages or change its sync settings with crawl.update(...), or PATCH /v1/crawls/:id.

    crawl.cancel(), or POST /v1/crawls/:id/cancel, stops it. Queued pages are released and no more are captured. Canceling a crawl that is already closed returns its status.

    Credit usage

    Every page Expand renders costs 1 credit, the same as a Fetch. Pages the crawl opens only to follow their links, and never keeps, cost nothing. See Pricing & Usage.

    Crawl errors

    On top of the standard errors, crawl requests use their own shape, { error: { code, message, retryable } }. In the SDK they are a CrawlClientApiError with code, status, and retryable. The errors relevant to creating, updating, streaming, and reading a crawl are:

    codeHTTPRetryable
    idempotency_key_conflict409No
    crawl_run_conflict409No

    idempotency_key_conflict

    The idempotency key was already used for a different request.

    HTTP 409Not retryable
    { "error": { "code": "idempotency_key_conflict", "message": "…", "retryable": false } }

    Use a new key for a new request.

    crawl_run_conflict

    A closed crawl cannot start another run, or the run command cannot be replayed. Create a new crawl for a new run.

    HTTP 409Not retryable

    crawl_blocked

    A source does not resolve to a public address.

    HTTP 422Not retryable
    { "error": { "code": "crawl_blocked", "message": "…", "retryable": false } }

    budget_exhausted

    Your account cannot pay for the crawl.

    HTTP 429Not retryable
    { "error": { "code": "budget_exhausted", "message": "…", "retryable": false } }

    Add credits, then create the crawl again.

    not_found

    No crawl with that ID exists for your organization.

    HTTP 404Not retryable
    { "error": { "code": "not_found", "message": "…", "retryable": false } }

    invalid_offset

    The from offset or snapshot cursor is invalid. Use the offset returned by the matching endpoint.

    HTTP 400Not retryable

    crawl_stream_offset_expired

    The requested event history is no longer retained. Replay from offset -1, or read the current catalog with crawl.snapshot() and resume from its final offset.

    HTTP 410Not retryable

    capture_expired

    A capture ID is beyond its retained horizon. The full content for that capture is no longer available.

    HTTP 410Not retryable

    temporarily_unavailable

    Expand could not serve the request right now.

    HTTP 503Retryable
    { "error": { "code": "temporarily_unavailable", "message": "…", "retryable": true } }

    Try again. The SDK retries it for you.

    Next steps

    • WebSync: keep a crawl running and get every change as an event.
    • TypeScript SDK: Crawl and WebSync: every option, method, and event field.
    • Pricing & Usage: what a crawl costs, and how credits are billed.
    • Errors & retries: the standard errors every request can return.
    PreviousWorking with snippets
    NextWebSync: Overview

    On This Page

    How a crawl finds pagesStarting a crawlWhat you get backCrawl statesWhy a crawl stops earlyChanging or stopping a crawlCredit usageCrawl errorsidempotency_key_conflictcrawl_run_conflictcrawl_blockedbudget_exhaustednot_foundinvalid_offsetcrawl_stream_offset_expiredcapture_expiredtemporarily_unavailableNext steps
    'page.added'
    )
    continue
    // Process event.page, then persist event.offset.
    }
    # Mirrors the site into a folder of Markdown files. The SDK and API stream events instead.
expandai crawl https://example.com/products ./products
    # Start the crawl. The response lists one crawl per domain.
curl https://api.expand.ai/v1/crawls \
  -H "x-expand-api-key: $EXPAND_API_KEY" \
  -H "x-idempotency-key: catalog-import-2026-09" \
  -H "Content-Type: application/json" \
  -d '{"sources":["https://example.com/products"],"formats":["markdown","stateJson"]}'

# Read its events from the beginning.
curl "https://api.expand.ai/v1/crawls/$CRAWL_ID/events?offset=-1" \
  -H "x-expand-api-key: $EXPAND_API_KEY"
    page.failed
    A page could not be captured. It carries a reason.
    paused
    Waiting, for example because your account cannot pay for the next pages.
    canceledStopped by you.
    failedCould not finish, for example because every source failed and no page was kept.
    source-partial
    A sitemap could not be read in full.
    Check that the sitemap is reachable.
    trap-guardURLs that look like crawl traps were skipped: very long URLs, more than 20 query parameters or path segments, or a path segment repeated 3 or more times.Usually nothing. Add exclude patterns if real pages were skipped.
    origin-errorsThe site returned too many errors in a row, so the crawl backed off.Try again later.
    billingThe crawl paused because your account cannot pay for the next pages.Add credits. A crawl still paused for billing after 24 hours is canceled.
    crawl_blocked422No
    budget_exhausted429No
    not_found404No
    invalid_offset400No
    crawl_stream_offset_expired410No
    capture_expired410No
    temporarily_unavailable503Yes