How to tell an Expand failure from a dead URL, which errors to retry, and what you are charged for.
Check the HTTP status and the error body together:
A 200 can still report an origin failure. A non-2xx can be an API or capture error, or an origin refusing to serve a file. Browser pages and files report origin errors differently below.
Nothing retries a dead URL by accident. Every response, success or error, carries an x-expand-trace-id header. Include it when you contact support.
This page is the guide. Every status code, error type, and example is in API reference: Errors.
When a request fails with a non-2xx response, the body carries a _tag that names the error and, for errors that might be temporary, a retryable boolean:
Errors that are unambiguous from their status alone (400, 401, 403, 404, 409, 410, 413) omit retryable. Treat a missing field on those as false.
When a browser page's origin fails, Fetch returns 200 with a FetchOriginError instead of, or next to, the page. A dead page URL, a broken certificate, or a site that answers 404 is a 200 with an origin error attached. A file's origin can instead return 404, 410, or 502 with FetchOriginHttpError. The fields, the headers on POST /v1/fetch, and examples are in When the origin fails.
Crawl uses its own shape: { "error": { "code", "message", "retryable" } }. See Crawl errors.
Not every URL can be fetched. The domain may not resolve, the certificate may be broken, nothing may be listening, or the origin may answer with a 404. Those are the origin's failures, not Expand's, and Fetch reports them as such.
POST /v1/fetch/json returns a typed FetchOriginError instead of a document when the origin was never reached:
{
"_tag": "FetchOriginError",
"url": "https://checkout.deadstartup.example/pricing",
"kind": "dns",
"upstreamError": "net::ERR_NAME_NOT_RESOLVED",
"retryable": false
}When a browser page's origin did answer with an error status, the captured content still comes back with the outcome attached, because an error page is often the answer you needed:
{
"meta": { "url": "https://site.example/removed-page", "title": "Page not found" },
"markdown": "# Page not found\n\n…",
"json": [],
"originError": {
POST /v1/fetch answers in Markdown, so the same facts ride the x-expand-origin-error, x-expand-origin-error-detail, and x-expand-origin-error-retryable response headers. When no document exists, the body is a short Markdown error document an agent can read directly.
A file has no error page to render. If its origin refuses it, Fetch returns a non-2xx FetchOriginHttpError: 404 or 410 for a missing or removed file, and 502 for other origin statuses. See Document errors.
For browser origin outcomes, kind says what went wrong and retryable says whether trying again could help. Network-error verdicts use the exact Chromium code; HTTP-error verdicts use the origin status. Every kind, field, and header is in API reference: Errors.
Trust retryable:
| Outcome | Retry? |
|---|---|
200 with FetchOriginError, retryable: true | Yes, after a delay. The condition may clear. |
200 with FetchOriginError, retryable: false | No. The URL is dead. Fix the URL or drop it. |
200 with no origin error | Nothing to retry. |
200 carrying an origin error as a transport failure. It is a completed answer about a URL that could not serve content.When a response carries a Retry-After header (429, 503, and 529 do), wait that many seconds. Otherwise back off exponentially with jitter, up to about 32 seconds. Hitting a limit is the most common reason for a 429. What the limits are is on Limits.
The TypeScript SDK follows the same rules. It retries when an error says retryable: true, up to maxRetries (default 2) within one timeoutMs budget (default 60000), which covers the retries and the waits between them.
It never retries the Batched Fetch 409 conflicts, the 502 page errors, or the 504 capture timeout, because repeating them reaches the same answer. It does retry 529 FetchCapacityTimeout and 503 FetchDocumentProcessingUnavailable with normal backoff, and 502 FetchOriginHttpError for a document. It does not retry 403, 413, 415, a 404 or 410 document refusal, or a 200 origin outcome.
Older TypeScript SDK versions expose an unrecognized 404 or 410 body as raw JSON text. Upgrade the SDK if you read structured fields from the error body. Crawl calls surface errors as a CrawlClientApiError with code, status, and retryable. Options are in TypeScript SDK: Errors and retries.
You pay for delivered content, not for attempts. Charges follow the outcome, not the status code.
| Outcome | Billed |
|---|---|
| Page captured successfully | Yes, one credit |
Origin responded with an error status (400+) and content was captured | Yes, the content was delivered alongside the originError |
Origin unreachable, any network FetchOriginError kind | No |
| (403) |
The one case that surprises people is the second row: an origin's 404 or 500 page still contains content, such as a soft-404, a maintenance page, or a status dashboard, and Expand captured and returned it, so it costs a credit like any other page. If you do not want to pay for error pages, check originError before treating a response as useful.
Everything else is straightforward: if no content reached you, no charge was made. See Pricing & Usage for what each operation costs in credits.
ExpandClient throws for non-2xx API responses, including a file origin's FetchOriginHttpError. Inspect status and body on ExpandClientApiError to branch on the _tag. See the TypeScript SDK for the full error hierarchy.
import { ExpandClient, ExpandClientApiError, ExpandClientError } from '@expandai/sdk'
const expand = new ExpandClient()
try {
const result = await expand.fetchJson({ url: 'https://example.com' })
// Reached here means the API returned 200. Check result for an origin outcome.
} catch (error) {
if (error instanceof ExpandClientApiError) {
// A non-2xx response. `body` is the parsed error above. Branch on `_tag`.
console.log('Status:', error.status)
console.log('Tag:', (error.body
The SDK already retries for you, using the same retryable field. It never retries the 502 page errors, FetchNavigationFailed and FetchCaptureFailed, and it waits out Retry-After before retrying a 429. Anything that reaches your catch block has run out of attempts.
fetchJson resolves to either shape, so narrow before reading content. Only the origin-outcome variant carries a top-level _tag:
import { isFetchOriginError } from '@expandai/sdk'
const result = await expand.fetchJson({ url: 'https://example.com' })
if (isFetchOriginError(result)) {
// The origin was never reached, so there is no content.
console.log(result.kind, result.retryable)
} else if (result.originError) {
// The origin answered with an error status, but the page was captured.
console.log(result.originError.originStatusCode, result.markdown)
} else {
console.log(result.markdown)
}Check the status code first, then the body. On /v1/fetch, the origin outcome is in the response headers.
response=$(curl -s -D /tmp/expand-headers -w "\n%{http_code}" \
-X POST https://api.expand.ai/v1/fetch \
-H "x-expand-api-key: $EXPAND_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}')
status_code=$(echo "$response" | tail -n 1)
body=$(echo
If you are calling the API directly rather than through an SDK, retry only when the body says to,
and never sooner than Retry-After asks:
retryable from the error body. If it is false, stop. No retry can succeedRetry-After seconds (or the body's retryAfterSeconds, if present)Retry-After, so an
unjittered wait sends the whole population back at one instantA successful /v1/fetch responds with text/markdown, not JSON. Read it with response.text().
Only the error responses are JSON, and a failure from an intermediary (a proxy's own error page, an
empty body) may not be JSON either, so parsing it has to be allowed to fail.
async function fetchWithRetry(url: string, maxRetries = 5) {
let delay = 1000
for (let attempt = 0; attempt < maxRetries; attempt++) {
const response = await fetch('https://api.expand.ai/v1/fetch', {
method: 'POST',
headers: { 'x-expand-api-key': API_KEY, 'content-type': 'application/json' },
body: JSON.stringify({ url }),
})
// The success body is the Markdown document itself.
200 may carry an origin outcome. A non-2xx may be an API or capture error, or a file origin's FetchOriginHttpError refusal (404, 410, or 502)._tag, not on message text.retryable when present. For browser origin network errors, it is derived from the exact Chromium code. false means stop, and true means the condition may clear. For a file-origin FetchOriginHttpError, use the HTTP status because its body has no retryable field.Retry-After. The SDK also retries a file-origin 502 FetchOriginHttpError.403, 413, 502) or the conservative timeout fallback (). A deliberate 504 retry may still be appropriate, but each attempt re-runs a full capture.404 or 410 with FetchOriginHttpError for a file | No. The file is missing or gone. |
400, 401, 403, 413 | No. Fix the request, the credentials, the balance, or the URL. |
429 | Yes, following retryable and Retry-After. See Limits. |
500 | Yes when the body says retryable: true. Report it with the trace ID if it persists. |
502 with FetchNavigationFailed or FetchCaptureFailed | No. The page or the capture fails the same way every time, so another attempt reaches the same result. |
502 with FetchOriginHttpError for a file | Yes. The file's origin may serve it later; the SDK retries this outcome. |
503 | Yes, after Retry-After. Expand could not prove the origin or the page was at fault, so the failure counts as Expand's and may clear. |
504 with FetchCaptureTimeout | No automatic retry. A deliberate retry may still be appropriate. |
529 with FetchCapacityTimeout | Yes, after Retry-After. The capture never started. |
FetchBlocked| No |
FetchPageTooLarge (413) | No |
ServiceUnavailable (503) or InternalError (500) | No |
504180000 ms client timeout for browser captures or 3600000 ms for documents. Documents can process for up to 55 minutes.x-expand-trace-id on every response. It is what support needs, and it is present even when the body is not.400 errors.