A crawl that keeps going. Expand rechecks every page on the schedule you set and streams each change as an event.
WebSync is a crawl that keeps going. Add sync to a crawl and Expand rechecks every page on the schedule you set, and streams each change as an event, in order, with an offset you can resume from. Use it to keep a site in sync with your database or your file system.
import { ExpandClient } from '@expandai/sdk'
const expand = new ExpandClient()
const [crawl] = await expand.crawls.create(
# Keeps a folder of Markdown files current. The SDK and API stream events instead.
expandai sync https://example.com/pricing ./pricing --every 24hcurl https://api.expand.ai/v1/crawls \
-H "x-expand-api-key: $EXPAND_API_KEY" \
-H "x-idempotency-key: pricing-watch" \
-H "Content-Type: application/json" \
-d '{"sources":["https://example.com/pricing"],"sync":{"freshness":"24h"}}'
# Wait for new events after the offset you saved.
curl "https://api.expand.ai/v1/crawls/$CRAWL_ID/events?offset=$SAVED_OFFSET&live=long-poll" \
-H "x-expand-api-key: $EXPAND_API_KEY"The sync settings:
| Setting | Definition | Default |
|---|---|---|
freshness | How often every page is checked for changes, such as 24h or 7d. How each check is made is up to the crawl. | Required |
renderEvery | How often a page is rendered in full, even when a check saw no change. | 7d |
A synced crawl keeps its stream open. crawl.events() returns once you have caught up, and live: true keeps waiting for new events. Everything else about a crawl, such as how it finds pages, its states, and its errors, is on Crawl.
Some parts of a page always change: a rotating banner, a timestamp, a view count. changePolicy tells WebSync which parts to leave out of the comparison, so they do not trigger a change:
changePolicy.ignore removes the matching parts of the page before comparing.changePolicy.volatile replaces them with stable markers.Both take CSS selectors, and both leave the saved page content as it was.
A page.changed event means a page differs from its last capture.
for await (const event of crawl.events({ from: savedOffset, live: true })) {
if (event.type !== 'page.changed') continue
console.log(event.url, event.changed) // { content: true, stateJson: false, extraction: false }
console.log(event.diff.markdown)
}curl "https://api.expand.ai/v1/crawls/$CRAWL_ID/events?offset=$SAVED_OFFSET&live=long-poll" \
-H "x-expand-api-key: $EXPAND_API_KEY| Field | Definition |
|---|---|
changed | Which parts changed: content, stateJson, or extraction. |
diff.stateJson | Each changed State JSON path with its old and new value, as { from, to }. A price that went from 49 to 59 shows up here as one path. |
diff.markdown | A diff of the Markdown. |
A page.removed event means a page is gone. WebSync does not decide this from one bad check. It sends the event only after at least three checks agree.
| Field | Definition |
|---|---|
evidence.kind | Why: http-404, http-410, content-gone, or redirected-away. |
evidence.checks | The checks behind the decision, each with its captureId, observedAt, HTTP status, and finalUrl. |
Every capture of a page is kept as a version. Read them newest first:
for await (const version of expand.pages(event.pageId).versions()) {
console.log(version.observedAt, version.http.status, version.contentSha256)
}curl https://api.expand.ai/v1/pages/$PAGE_ID/versions \
-H "x-expand-api-key: $EXPAND_API_KEY"Each version has its captureId, observedAt, the stream offset it was published at, a contentSha256 of its content, the HTTP status and finalUrl, a quality of complete, best-effort, or degraded, and the page meta. Pass at with a timestamp to get the version that was current at that moment.
To compare any two captures of a page, not only neighbours:
const diff = await expand.pages(pageId).diff({ from: olderCaptureId, to: newerCaptureId })curl "https://api.expand.ai/v1/pages/$PAGE_ID/diff?from=$OLDER_CAPTURE_ID&to=$NEWER_CAPTURE_ID" \
-H "x-expand-api-key: $EXPAND_API_KEY"It returns the same diff as a page.changed event.
A check is free. Every page Expand renders costs 1 credit, the same as a Fetch. budget.rendersPerDay caps what a synced crawl can spend in a day. See Pricing & Usage.
budget.rendersPerDay | The most pages rendered in a day. | None |
diff.truncated | true when the diff was too large to send in full. |
captureId, previousCaptureId | The new capture and the one before it. Read either in full with expand.captures(id). |
evidence.sitemapDisappeared | true when the page also dropped out of the site's sitemap. |
previousCaptureId | The last capture of the page while it existed. |