# Error recovery playbook (real-world failure modes)

Nukez is designed to surface real production failures: payment propagation delays, token expiry, URL expiry, and transient signing errors.

This document is a practical "what to do next" map.

## Payment & x402

### `/v1/storage/request` returns HTTP 402

This is expected for the x402 payment protocol.

What to do:

1) Parse the JSON body for:
- `pay_req_id`
- `pay_to_address`
- `amount_sol` (or `amount_lamports`) for Solana; `amount_raw` (or `amount_wei`) for EVM
- `pay_network` (e.g., `solana`, `monad`)
- `pay_asset` (e.g., `SOL`, `MON`)
- network metadata (devnet / mainnet / testnet)

2) Execute the payment transfer **exactly** as instructed on the specified chain.

3) Call `POST /v1/storage/confirm` with:
- Header: `X402-TX: <tx_sig>`
- Body: `{"pay_req_id":"..."}`

### `/v1/storage/confirm` returns 402 even after successful payment

This usually means the `X402-TX` header is missing or malformed.

**Common mistake:** Putting `X402-TX` in the request body instead of as an HTTP header.

**Wrong:**
```json
{
  "pay_req_id": "abc123",
  "X402-TX": "5qZUNVPM..."
}
```

**Also wrong:** Appending to body as text:
```
{"pay_req_id":"abc123"}
X402-TX: 5qZUNVPM...
```

**Correct:** Send `X402-TX` as an HTTP header separately from the body:

```http
POST /v1/storage/confirm
Content-Type: application/json
X402-TX: 5qZUNVPMpqubdqBF8LMW6tEZEaPReKy7B4XfWRNwS7UMEZeHV1YU3cma5wPaPsddjaCxEndk5T6nATYZM4YUyKE

{"pay_req_id":"abc123"}
```

**For agents using HTTP tools:** If your tool has a `headers` parameter, use it:

```json
{
  "method": "POST",
  "url": "https://api.nukez.xyz/v1/storage/confirm",
  "headers": {
    "Content-Type": "application/json",
    "X402-TX": "<your_tx_signature>"
  },
  "body": {"pay_req_id": "<pay_req_id>"}
}
```

**Fallback if your tool cannot set headers:** Include `tx_sig` in the body:

```json
{
  "method": "POST",
  "url": "https://api.nukez.xyz/v1/storage/confirm",
  "body": {"pay_req_id": "<pay_req_id>", "tx_sig": "<your_tx_signature>"}
}
```

### `/v1/storage/confirm` returns 402 / `tx_not_found`

This usually means propagation delay -- the transaction has not reached the indexer yet. This error applies to both Solana and EVM chains, but with different retry characteristics.

**Solana:** Propagation delay is typically 1-5 seconds. Transactions are confirmed quickly once they land on-chain.

**EVM (Monad):** Block confirmation time varies by chain. Monad targets fast block times, but finality may still require multiple block confirmations. Initial delay before first retry should be longer than for Solana.

Retry strategy:
- Solana: exponential backoff starting at 2s: 2s, 4s, 8s, 16s, 32s
- EVM: exponential backoff starting at 5s: 5s, 10s, 20s, 40s (allow for block confirmation)
- Add jitter (+/- 20%) if you have multiple agents
- Cap retries (e.g., 5-7 attempts)

### `/v1/storage/confirm` returns 402 / `payment_verification_failed`

This usually means:
- wrong transaction signature / hash
- wrong amount (insufficient payment)
- payment sent to the wrong address
- (EVM) transaction reverted or ran out of gas

Actions:
- Fetch and re-check the tx on the relevant chain explorer (amount + destination).
- For EVM: verify the transaction was not reverted and gas was sufficient.
- If wrong amount/address, the safest next step is to re-run `request` and pay again.

### `/v1/storage/request` or `/v1/storage/confirm` returns 409

Common meaning: idempotency conflict.

Actions:
- If you reused the same `Idempotency-Key` with a different request body, use a fresh key.
- If the body is identical, retrying with the same key is safe.

## Payment rule failures

### 409 `PAYMENT_ALREADY_ALLOCATED` from `/v1/storage/confirm`

One payment transaction buys exactly one pay request. This transaction already bought another `pay_req_id`.

Actions:
- If you are retrying the confirm for the pay request that this transaction originally bought, the retry is idempotent; check that you sent the same `pay_req_id`.
- If you meant to buy a second locker, make a second payment and confirm it with its own `pay_req_id`.

### 402 `PAYMENT_OUTSIDE_QUOTE_WINDOW` from `/v1/storage/confirm`

The transaction was not made for this quote: it was sent before the quote was computed (more than 300 seconds earlier) or long after it expired (more than 3600 seconds after `quote_expires_at`).

Actions:
- Request the quote first, pay it, then confirm. A quote lives 300 seconds; the window is generous but real.
- If your payment landed late, request a fresh quote and pay it again. The earlier transaction can be recovered only by the treasury owner.

### 422 `PAYMENT_OPTION_NOT_IN_QUOTE` from `/v1/storage/confirm`

The `pay_asset` / `pay_network` you echoed was not one of the legs offered in that quote's `accepts[]`.

Actions:
- Echo the leg you actually paid, exactly as it appeared in `accepts[]`. If you paid the default leg, omit both fields.

### 401 `CONFIRM_INTENT_REQUIRED` or `CONFIRM_INTENT_INVALID` from `/v1/storage/confirm`

You sent `operator_pubkey` (binding an operator at confirm) without a valid `payer_signature`.

Actions:
- Sign the exact string in the response's `sign_this` field with the payer key (`nukez/confirm/v1|<pay_req_id>|<tx>|<operator_pubkey>`) and resend it as `payer_signature` (base58 for Ed25519, `0x`-hex EIP-191 for EVM).
- Or confirm without `operator_pubkey` and add the operator later with `POST /v1/lockers/{locker_id}/operators`.

### The asset you wanted is missing from `accepts[]`

Prices come from live feeds only. When an asset's feed is unavailable, or its two feeds disagree, the gateway omits that asset from the quote rather than pricing it from a constant.

Actions:
- Pay with another offered leg, or request a fresh quote a little later.

## Signed envelope failures

Signed-envelope failures are usually "you signed the wrong bytes".

### 422 from a signed-envelope endpoint (e.g., /v1/storage/signed_provision)

Most common cause: missing X-Nukez-Envelope and/or X-Nukez-Signature headers.
This happens when you construct the envelope correctly but then make an HTTP request without attaching the headers.
Diagnosis: If you used a signing helper and it returned successfully, but you still got 422, you almost certainly forgot to pass the headers to your HTTP tool.
Fix:
If your signing tool/helper returned something like:
```json
{
  "headers": {
    "X-Nukez-Envelope": "eyJ2IjoxLCJsb2NrZXJfaWQiOi4uLn0",
    "X-Nukez-Signature": "5QzPmN8xK..."
  },
  "body_canonical": "{\"receipt_id\":\"abc123\",\"tags\":[]}"
}
```
Then your HTTP request MUST include those headers:
```json
{
  "method": "POST",
  "url": "https://api.nukez.xyz/v1/storage/signed_provision",
  "headers": {
    "X-Nukez-Envelope": "eyJ2IjoxLCJsb2NrZXJfaWQiOi4uLn0",
    "X-Nukez-Signature": "5QzPmN8xK...",
    "Content-Type": "application/json"
  },
  "body_bytes": "{\"receipt_id\":\"abc123\",\"tags\":[]}"
}
```

A related failure looks similar but returns 401 instead of 422: your signature covered different body bytes than what you sent (`401 ENVELOPE_BODY_MISMATCH`).
Actions:
   1. Ensure the request includes both headers: X-Nukez-Envelope and X-Nukez-Signature.
   2. Ensure the body bytes match the body_sha256 in the envelope:
      - If using JSON, canonicalize it first, hash those exact bytes, and send those exact bytes over the wire.
      - Do not let an HTTP client/tool re-serialize a JSON object differently than what you hashed.
   3. Rebuild the envelope using the exact body you will send, then retry once.

**Note:** Signed envelope auth applies to ALL authenticated endpoints, not just POST:
- **GET** requests (list files, get file URLs, recompute-verify): the headers are required, and `body_sha256` is the SHA-256 of the empty string.
- **DELETE** requests: the headers are required, and `body_sha256` is the SHA-256 of the empty string.
- **POST** requests: the headers are required, and `body_sha256` covers the request body.

**How to read the status code when headers are missing (the divergence is intentional):** on most signed routes — including `/v1/storage/attest`, `/v1/attest/push`, and `/v1/storage/recompute-verify` — the two headers are declared required, so a request with the headers entirely missing returns `422 VALIDATION_ERROR` before any auth logic runs, and the 401 family appears only when the headers are present but fail verification. The confirm endpoints are the deliberate exception: `/v1/files/confirm` and `/v1/files/confirm-batch` answer missing headers with `401 SIGNED_ENVELOPE_REQUIRED` once enforcement is active. So a 401 on a GET such as list-files means your headers were present but invalid (bad signature, expired window, wrong ops, or a receipt mismatch); if you forgot the headers entirely, you would have received a 422 on those routes.

### 401 `BAD_SIGNATURE`

Check:
- You used the payer's correct key (Ed25519 for Solana, secp256k1 for EVM).
- `X-Nukez-Signature` is base58 of the 64-byte Ed25519 signature (Solana) or hex-encoded secp256k1 signature (EVM).
- `X-Nukez-Envelope` is base64url **without padding**.

### 401 `ENVELOPE_BODY_MISMATCH`

Check:
- You hashed the exact bytes you sent over the wire.
- If you send JSON, serialize it canonically (sorted keys, compact separators, `ensure_ascii=False` so UTF-8 goes over the wire unescaped), then hash that byte string.
- Don't let your HTTP client re-serialize JSON differently than what you hashed.

### 403 `MISSING_OPS`

Check:
- Your `envelope.ops` includes the required op(s) for the endpoint.
- Use `x-nukez-ops` from `/openapi.json` as the source of truth.

### 401 `ENVELOPE_EXPIRED`

Check:
- `exp` is in the future.
- Your clock isn't badly skewed.
- Keep `exp - iat` short (<= 60s) but not too short for slow networks.

### 400 `ENVELOPE_TTL_TOO_LONG`

The gateway rejects any envelope whose TTL (`exp - iat`) exceeds 300 seconds. Use a short TTL and sign the envelope immediately before sending the request it authorizes. For confirm calls specifically, sign the envelope at confirm time, after the upload finishes — a long upload must re-sign rather than reuse an envelope that was built before the upload started.

## Operator delegation failures

### 403 from `POST /v1/lockers/{locker_id}/operators` or `DELETE .../operators/{pubkey}`

Meaning: the signed envelope was verified, but the signer is not the locker owner.

Only the locker owner (the original payer) can add or remove operators. Operators themselves cannot manage other operators.

Actions:
- Ensure the envelope is signed by the payer key that provisioned the locker, not by an operator key.
- Verify `ops` includes `locker:admin`.

### 403 from file endpoint when using an operator key

Meaning: the operator key is not registered on this locker.

Actions:
- Confirm the operator was added via `POST /v1/lockers/{locker_id}/operators` before use.
- Verify the operator's Ed25519 public key matches what was registered.
- If the operator was removed, re-add it (owner action).

### Operator provision-time registration

If you pass `operator_pubkey` at provision time (`POST /v1/storage/signed_provision`) and get no error, the operator is registered. If provision fails, the operator was not registered — fix the provision error first.

## Confirm upload failures

`POST /v1/files/confirm` takes `receipt_id` and `filename` as query parameters, the POST body is the literal two characters `{}`, and the request must carry a payer-signed envelope with the `locker:write` operation — the same authority as the create-file step. The envelope binds the method, path, query string, and the canonical body `{}`, and its `receipt_id` must match the query `receipt_id`. Enforcement is being rolled out in stages, but the signed request is the contract.

### 422 from `POST /v1/files/confirm`

Common causes:
- `receipt_id` or `filename` is missing from the query string (they are query parameters, not body fields).

Actions:
- Send both values in the query string, for example `POST /v1/files/confirm?receipt_id=abc123&filename=notes.txt`, with `{}` as the body.

### 401 `SIGNED_ENVELOPE_REQUIRED` from `POST /v1/files/confirm`

The envelope headers were missing while enforcement is active. Unlike the attest and recompute-verify routes (where entirely missing headers produce a 422), the confirm endpoints deliberately answer missing headers with this 401.

Actions:
- Sign a `locker:write` envelope at confirm time — after the upload completes, since the gateway rejects envelopes older than 300 seconds — and attach `X-Nukez-Envelope` and `X-Nukez-Signature`.
- Make sure the envelope's `receipt_id` equals the query `receipt_id`, and that `body_sha256` is the SHA-256 of exactly the two characters `{}`.

### 404 file not found on confirm

The named file does not exist in the locker manifest.

Actions:
- Verify the file was created via `POST /v1/lockers/{locker_id}/files` before confirming.
- Ensure the filename matches exactly (case-sensitive).

### Hash mismatch on confirm (409)

If the gateway computes a SHA-256 hash that differs from what was expected, the upload bytes were corrupted or incomplete.

Actions:
- Re-upload the file bytes to the upload_url.
- Re-confirm after successful upload.

## Fetch-and-store failures

### 422 from `POST /v1/files/fetch-and-store`

Common causes:
- Invalid or unreachable `source_url`.
- SSRF protection blocked the target URL (internal/private IP ranges).

Actions:
- Verify the source_url is a publicly accessible HTTPS URL.
- Do not use localhost, private IPs, or internal hostnames.

### Timeout on fetch-and-store

Large files may exceed the gateway's fetch timeout.

Actions:
- For large files, upload directly via the create_file → PUT upload_url path instead.

## Sandboxed app uploads (ChatGPT/Claude-style runtimes)

In proxied runtime sandboxes, direct local path uploads can fail even when the file exists.

### Deterministic detached upload path (recommended)

Use upload sessions so bytes move out-of-band from the model context:

1) `POST /v1/lockers/{locker_id}/upload-sessions`
- include `receipt_id` and file metadata
- receive `session_id`, `upload_token`, and per-session endpoints

2) `POST /v1/upload-sessions/{session_id}/files/{file_id}/parts`
- send base64 parts using `X-Upload-Token` (or bearer token)
- avoid direct `/mnt/data` path assumptions in sandbox tools

3) `POST /v1/upload-sessions/{session_id}/finalize`
- assembles bytes, confirms upload, runs integrity checks
- returns `viewer_url` + UI button contract

4) `GET /v1/upload-sessions/{session_id}`
- poll terminal status (`complete|partial|failed`)

### Structured recovery fields

Upload-related errors include:
- `details.layer` (`app_orchestrator|sandbox_runtime|mcp|gateway|provider|viewer`)
- `details.recovery_hint`
- `details.next_best_method`

Use these fields directly for automatic fallback decisions.

### Storage purchase guardrail

If a valid `receipt_id` already exists in context, reuse it.
Do not call storage purchase endpoints again unless the user explicitly requests a new purchase.

## Short URL failures (upload_url / download_url)

All upload_url and download_url values are short URLs: `https://api.nukez.xyz/f/{token}`. The gateway decodes the Ed25519-signed token and either proxies or 307-redirects to the underlying storage provider. This redirect is transparent to standard HTTP clients.

### 403 from a short URL

The URL token has expired (default TTL: 30 minutes), the signature is invalid, or the request used a different HTTP method than the one the token was signed for. Short URL tokens are bound to exactly one method: a download (GET) token cannot be replayed as a PUT or DELETE, and the resolver rejects the mismatch with `method_not_allowed_for_token`.

Actions:
- Refresh by calling `GET /v1/lockers/{locker_id}/files/{filename}` (or re-running `create_file`).
- Then use the new `upload_url`/`download_url`, and use each URL only with the method it was minted for (upload_url for PUT, download_url for GET).

### 404 from a short URL

The file does not exist in the locker, or the URL token references a file that was deleted.

Actions:
- Call `GET /v1/lockers/{locker_id}/files` (list_files) to see what exists.
- Re-create the file with `POST /v1/lockers/{locker_id}/files`.

### Upload fails with 403 due to Content-Type mismatch

If the API created the file with a locked content type, your PUT must match.

Actions:
- Use the `content_type` you provided at create-time.
- If the API returned `upload_headers`, include them exactly.
- Note: the 307 redirect preserves your Content-Type header.

### After a successful PUT, the manifest has no content hash

Which side records the hash depends on which upload path you used. A proxied PUT (`?proxy=1`) streams the bytes through the gateway, which computes the SHA-256 itself, records it into the manifest, and reports the outcome in the `X-Nukez-Content-Hash-Recorded` response header (`true` or `false`). A plain PUT is answered with a 307 redirect to a provider signed URL, and that redirect carries `X-Nukez-Confirm-Required: true` — meaning the gateway never saw your bytes, so you must still call `POST /v1/files/confirm` (or `/v1/files/confirm-batch`) with a payer-signed `locker:write` envelope after the redirected upload completes.

Actions:
- If you uploaded through the redirect path, call the confirm endpoint.
- If you uploaded through the proxy and `X-Nukez-Content-Hash-Recorded` came back `false`, call the confirm endpoint to record the hash explicitly.

### Verifying short URL tokens independently

If you want to verify a short URL token was issued by this Nukez gateway:
1) Fetch the public key: `GET /v1/short-url/verify-key` -> `verify_key_hex`
2) Base64url-decode the token, split payload (all but last 64 bytes) from signature (last 64 bytes)
3) Verify Ed25519 signature using the public key

This requires no shared secret and no Nukez account. One honest caveat: the verify key is served by the same gateway that mints the tokens, so a fully trust-free verifier must pin the key out of band — a gateway that chose to tamper could serve both a forged token and a matching key. Also note that this key signs short-URL tokens only; it is a different key from `receipt_signer_pubkey` (served on `GET /v1/receipts/{receipt_id}`), which is the key that verifies `receipt_sig` and `manifest_signature`.

## Retired endpoints

### 410 `endpoint_retired` from `POST /v1/attest/simulate`

Oracle simulation was retired with the Switchboard integration. Nothing calls back any more.

Actions:
- Read the proof on `GET /v1/storage/verify`, or download `GET /v1/storage/evidence-bundle?receipt_id=...` and verify it independently with the pinned keys from `GET /v1/keys`.

## Transient 5xx errors

### 500 `IAM_SIGNBLOB_PERMISSION_DENIED` (or similar)

Meaning: the service can't sign storage URLs due to missing IAM permissions.

Actions:
- This is not an agent-recoverable problem. Report it.

### Other 5xx

Actions:
- Retry with exponential backoff + jitter.
- Prefer retries for idempotent operations (GETs, confirms with the same tx, etc.).
- For non-idempotent operations (e.g., creating new files with server-generated names), retry carefully and de-duplicate by listing files.