Batched Fetch

Jump to: Start a run · Poll results · API Reference

Batched Fetch is API/SDK-only today. There is no CLI command or MCP tool for starting a batch.

Start a run

Send all the URLs in one POST. The call returns once Hatchet durably accepts the run command; billing reservation, lifecycle projection, and browser captures continue in the background.

urls is required and must be non-empty. include is optional, and the only public browserConfig field at start is scrollFullPage. Unknown top-level keys are rejected.

• Every submitted URL uses the same shared settings. The start body does not accept per-URL settings.
• Duplicate URLs are deduped by resolved URL, so the same page is captured once.
• Production callers should authenticate with an API key. Anonymous sessions cannot start batched runs.

Keep the run ID. Every status check and every page of results is read back through it.

An immediate poll can return QUEUED with no results while the accepted command is being projected. This is expected. Billing is checked authoritatively during that initialization, so a run can later become FAILED even though the start request was accepted.

Safe retries

When x-idempotency-key is omitted, the API generates a unique key. Send a stable key when the caller needs to retry a separate request or recover after a process restart. Retrying the same payload with the same explicit key is safe. The API returns a successful response with the same run ID and does not create or charge for another run.

Only reusing the same key with a different request body returns 409 BatchedIdempotencyConflict:

Treat payload_mismatch as a client bug rather than polling a run created for different input.

• Keys are scoped to your organization and may be up to 255 characters.
• The API generates a key when omitted. Persist or deterministically reconstruct an explicit key when deduplication must survive separate calls or process restarts.
• Include a logical period in recurring-job keys, such as fastenersolutions:2026-07:<digest>.

Poll results

Read status and results back with the run ID. Page through results with limit and offset.

Keep polling while batchedStatus is QUEUED or RUNNING. Stop once it reaches a terminal status.

• Use a short delay or backoff between polls instead of a tight loop.
• Handle API errors from the poll call the way you handle any request.
• FAILED or CANCELLED means stop polling the run.
• After the run is terminal, inspect each result's status before using it.

Status model

batchedStatus describes the whole run. The top-level status describes the current page of results, and each result has its own persisted status.

• QUEUED means the run was accepted but may not be executing yet.
• RUNNING means work is still in progress.
• COMPLETED on batchedStatus means the run is done, not that every URL produced Markdown.
• FAILED and CANCELLED are terminal.
• A page reads COMPLETED only when its rows are stable and requested content extraction succeeded. Requested content can legitimately be empty.
• A page can read FAILED after a completed capture if snapshot content could not be read or rendered. Successful siblings on that page remain usable.
• Failed extraction outcomes are cached for one minute. Re-poll the same failed page with backoff to retry extraction without starting or charging for another browser capture.
• A page can read COMPLETED while the whole run is still RUNNING. Keep the two fields separate in your code; do not collapse them into one status.

Response shape

A poll returns the run status, pagination metadata, and a page of per-URL results.

Handle each result independently. A batch can finish even when one URL was blocked, failed, or redirected. Use results[].status rather than missing Markdown to identify failed items. A blocked result remains SUCCEEDED and carries data.blocked; fully populated siblings remain usable.

If no poll include.markdown is provided, Batched Fetch normalizes results to Markdown by default. For the exact per-result schema, see the API Reference.

error and originError

An item carries exactly one of data or error. originError sits outside that pair and can accompany the captured data.

A URL whose origin could not be reached — DNS failure, broken TLS, refused connection, connection timeout — is not a batch failure. The item reads SUCCEEDED at the item level, because Expand did its job and established that the URL has nothing to serve, and carries a FetchOriginError in error with no data. This replaces the bare URL-only stub these URLs used to produce, which explained nothing.

An origin that answered with a 404 or 500 did produce a snapshot, so that item keeps its data and reports the status in originError instead.

kind is one of dns, tls, connectionRefused, connectionReset, connectionClosed, addressUnreachable, emptyResponse, connectionTimeout, or http, and retryable tells you whether another attempt could plausibly succeed based on the exact Chromium network code. See Error Handling for the full taxonomy. Unreachable URLs are not billed; captured error pages are — see Pricing & Usage.

Pagination

Batched results are paginated. Fetch page 1 with offset=0, then keep increasing offset by limit while pagination.hasMore is true.

• limit defaults to 10 and accepts 1 through 100.
• offset defaults to 0 and must be non-negative.
• pagination.total is the number of available result rows for the run.
• pagination.hasMore tells you whether to request the next page.

• Polling only the first page can miss later results.
• Page-level status can differ across pages while the run is still active.

Partial failures

Batched Fetch is item-oriented. Treat the batch as a container for many Fetch attempts, then inspect each result before using it.

• Blocked target pages can appear as per-item data.blocked, for example blockedType: "botProtection".
• An unreachable origin appears as a per-item error with _tag: "FetchOriginError" and no data. Read its retryable field before scheduling a re-run: false means the URL is dead, not that the batch went wrong.
• An origin that answered with a status of 400 or above keeps its data and reports the status in originError.
• data.response.originStatusCode helps classify each outcome, for example 403 on a blocked page.
• Pending rows can initially return only minimal data.response.
• A URL-only, unblocked result on a FAILED page can indicate an extraction failure; missing Markdown on a COMPLETED page can be legitimate empty content.
• Failed extraction outcomes are cached for one minute. Re-poll the same / page with backoff to retry reading existing capture artifacts without another capture or charge.

Retry a start request with the same key and payload after an ambiguous timeout. The API returns the same run ID if the first request already created the run.

Supported options

Polling rejects screenshot and appendix include fields with BatchedIncludeUnsupported when the original batch did not capture them. New batches always request Markdown and State JSON. Existing runs retain the formats captured when they were created.

Batched Fetch is narrower than single Fetch by design. Use it when many URLs and asynchronous collection matter more than every single Fetch artifact. For the full set of fields and nested defaults, see Include Options.

SDK examples

The TypeScript SDK wraps the same two calls: batched to start a run, getBatched to poll it.

Each successful result returns Main Markdown at data.markdown and State JSON at data.json without extra include configuration when extraction succeeds.

• When omitted, batched() generates one idempotencyKey and reuses it for the client's configured retry policy.
• Supply a stable idempotencyKey to deduplicate separate calls or calls made after a process restart.
• The same key and payload return the same run ID. Catch ExpandClientApiError with status 409 only for a different payload; its body contains existingRunId and reason: "payload_mismatch".
• The generated getBatched query params are strings today, so the examples use "10" and "0".

Operational notes

• Batched Fetch queues its child browser captures under the organization's browser concurrency rather than cancelling each other.
• Cancellation is cooperative. A successful cancel request returns CANCELLING while running items drain, or CANCELLED when no work remains; queued items are cancelled immediately.
• Higher tiers can have higher browser concurrency. Exact tier details live in Tiers and Pricing & Usage.
• Duplicate submitted URLs are deduped by resolved URL.
• Anonymous sessions cannot start batched runs; API-key auth is the normal production setup.
• There is no Batched-specific public rate-limit number to plan against today.

Common mistakes

• Polling once, seeing no results, and assuming the run failed.
• Stopping because a page status is COMPLETED while batchedStatus is still RUNNING.
• Ignoring pagination and reading only the first page.
• Assuming COMPLETED means every item has Markdown.
• Re-running URLs whose error reports retryable: false. The origin is unreachable; the batch did nothing wrong.
• Requesting screenshot or appendix while polling when the original batch did not capture it.
• Looking for a CLI or MCP batch command. There is none today.
• Expecting an automatically generated key to deduplicate separate calls or calls made after a process restart.
• Treating payload_mismatch as a recoverable duplicate instead of fixing key reuse.

Next steps

• Start Batched Fetch API Reference: exact create schema.
• Get Batched Fetch API Reference: exact poll and result schema.
• Include Options: the full include surface and limits.
• Browser Behavior: capture and scrollFullPage behavior.
• Highlights: search a single Fetch or existing snapshot.