# Error recovery playbook (real-world failure modes)

Nukez is designed to surface real production failures: payment propagation delays, token expiry, URL expiry, and transient signing errors.

This document is a practical "what to do next" map.

## Payment & x402

### `/v1/storage/request` returns HTTP 402

This is expected for the x402 payment protocol.

What to do:

1) Parse the JSON body for:
- `pay_req_id`
- `pay_to_address`
- `amount_sol` (or `amount_lamports`) for Solana; `amount_raw` (or `amount_wei`) for EVM
- `pay_network` (e.g., `solana`, `monad`)
- `pay_asset` (e.g., `SOL`, `MON`)
- network metadata (devnet / mainnet / testnet)

2) Execute the payment transfer **exactly** as instructed on the specified chain.

3) Call `POST /v1/storage/confirm` with:
- Header: `X402-TX: <tx_sig>`
- Body: `{"pay_req_id":"..."}`

### `/v1/storage/confirm` returns 402 even after successful payment

This usually means the `X402-TX` header is missing or malformed.

**Common mistake:** Putting `X402-TX` in the request body instead of as an HTTP header.

**Wrong:**
```json
{
  "pay_req_id": "abc123",
  "X402-TX": "5qZUNVPM..."
}
```

**Also wrong:** Appending to body as text:
```
{"pay_req_id":"abc123"}
X402-TX: 5qZUNVPM...
```

**Correct:** Send `X402-TX` as an HTTP header separately from the body:

```http
POST /v1/storage/confirm
Content-Type: application/json
X402-TX: 5qZUNVPMpqubdqBF8LMW6tEZEaPReKy7B4XfWRNwS7UMEZeHV1YU3cma5wPaPsddjaCxEndk5T6nATYZM4YUyKE

{"pay_req_id":"abc123"}
```

**For agents using HTTP tools:** If your tool has a `headers` parameter, use it:

```json
{
  "method": "POST",
  "url": "https://api.nukez.xyz/v1/storage/confirm",
  "headers": {
    "Content-Type": "application/json",
    "X402-TX": "<your_tx_signature>"
  },
  "body": {"pay_req_id": "<pay_req_id>"}
}
```

**Fallback if your tool cannot set headers:** Include `tx_sig` in the body:

```json
{
  "method": "POST",
  "url": "https://api.nukez.xyz/v1/storage/confirm",
  "body": {"pay_req_id": "<pay_req_id>", "tx_sig": "<your_tx_signature>"}
}
```

### `/v1/storage/confirm` returns 402 / `tx_not_found`

This usually means propagation delay -- the transaction has not reached the indexer yet. This error applies to both Solana and EVM chains, but with different retry characteristics.

**Solana:** Propagation delay is typically 1-5 seconds. Transactions are confirmed quickly once they land on-chain.

**EVM (Monad):** Block confirmation time varies by chain. Monad targets fast block times, but finality may still require multiple block confirmations. Initial delay before first retry should be longer than for Solana.

Retry strategy:
- Solana: exponential backoff starting at 2s: 2s, 4s, 8s, 16s, 32s
- EVM: exponential backoff starting at 5s: 5s, 10s, 20s, 40s (allow for block confirmation)
- Add jitter (+/- 20%) if you have multiple agents
- Cap retries (e.g., 5-7 attempts)

### `/v1/storage/confirm` returns 402 / `payment_verification_failed`

This usually means:
- wrong transaction signature / hash
- wrong amount (insufficient payment)
- payment sent to the wrong address
- (EVM) transaction reverted or ran out of gas

Actions:
- Fetch and re-check the tx on the relevant chain explorer (amount + destination).
- For EVM: verify the transaction was not reverted and gas was sufficient.
- If wrong amount/address, the safest next step is to re-run `request` and pay again.

### `/v1/storage/request` or `/v1/storage/confirm` returns 409

Common meaning: idempotency conflict.

Actions:
- If you reused the same `Idempotency-Key` with a different request body, use a fresh key.
- If the body is identical, retrying with the same key is safe.

## Signed envelope failures

Signed-envelope failures are usually "you signed the wrong bytes".

### 422 from a signed-envelope endpoint (e.g., /v1/storage/signed_provision)

Most common cause: missing X-Nukez-Envelope and/or X-Nukez-Signature headers.
This happens when you construct the envelope correctly but then make an HTTP request without attaching the headers.
Diagnosis: If you used a signing helper and it returned successfully, but you still got 422, you almost certainly forgot to pass the headers to your HTTP tool.
Fix:
If your signing tool/helper returned something like:
```json
{
  "headers": {
    "X-Nukez-Envelope": "eyJ2IjoxLCJsb2NrZXJfaWQiOi4uLn0",
    "X-Nukez-Signature": "5QzPmN8xK..."
  },
  "body_canonical": "{\"receipt_id\":\"abc123\",\"tags\":[]}"
}
```
Then your HTTP request MUST include those headers:
```json
{
  "method": "POST",
  "url": "https://api.nukez.xyz/v1/storage/signed_provision",
  "headers": {
    "X-Nukez-Envelope": "eyJ2IjoxLCJsb2NrZXJfaWQiOi4uLn0",
    "X-Nukez-Signature": "5QzPmN8xK...",
    "Content-Type": "application/json"
  },
  "body_bytes": "{\"receipt_id\":\"abc123\",\"tags\":[]}"
}
```

Second most common cause: your signature covered different body bytes than what you sent.
Actions:
   1. Ensure the request includes both headers: X-Nukez-Envelope and X-Nukez-Signature.
   2. Ensure the body bytes match the body_sha256 in the envelope:
      - If using JSON, canonicalize it first, hash those exact bytes, and send those exact bytes over the wire.
      - Do not let an HTTP client/tool re-serialize a JSON object differently than what you hashed.
   3. Rebuild the envelope using the exact body you will send, then retry once.

**Note:** Signed envelope auth applies to ALL authenticated endpoints, not just POST:
- **GET** requests (list files, get file URLs): Need headers, use empty string for body_sha256
- **DELETE** requests: Need headers, use empty string for body_sha256
- **POST** requests: Need headers, body_sha256 covers the request body

If you get 401 on a GET request, you likely forgot to attach the envelope headers.

### 403 `SIGNATURE_INVALID`

Check:
- You used the payer's correct key (Ed25519 for Solana, secp256k1 for EVM).
- `X-Nukez-Signature` is base58 of the 64-byte Ed25519 signature (Solana) or hex-encoded secp256k1 signature (EVM).
- `X-Nukez-Envelope` is base64url **without padding**.

### 403 `BODY_HASH_MISMATCH`

Check:
- You hashed the exact bytes you sent over the wire.
- If you send JSON, serialize it canonically, then hash that byte string.
- Don't let your HTTP client re-serialize JSON differently than what you hashed.

### 403 `MISSING_OPS`

Check:
- Your `envelope.ops` includes the required op(s) for the endpoint.
- Use `x-nukez-ops` from `/openapi.json` as the source of truth.

### 403 `EXPIRED_ENVELOPE`

Check:
- `exp` is in the future.
- Your clock isn't badly skewed.
- Keep `exp - iat` short (<= 60s) but not too short for slow networks.

## Operator delegation failures

### 403 from `POST /v1/lockers/{locker_id}/operators` or `DELETE .../operators/{pubkey}`

Meaning: the signed envelope was verified, but the signer is not the locker owner.

Only the locker owner (the original payer) can add or remove operators. Operators themselves cannot manage other operators.

Actions:
- Ensure the envelope is signed by the payer key that provisioned the locker, not by an operator key.
- Verify `ops` includes `locker:admin`.

### 403 from file endpoint when using an operator key

Meaning: the operator key is not registered on this locker.

Actions:
- Confirm the operator was added via `POST /v1/lockers/{locker_id}/operators` before use.
- Verify the operator's Ed25519 public key matches what was registered.
- If the operator was removed, re-add it (owner action).

### Operator provision-time registration

If you pass `operator_pubkey` at provision time (`POST /v1/storage/signed_provision`) and get no error, the operator is registered. If provision fails, the operator was not registered — fix the provision error first.

## Confirm upload failures

### 422 from `POST /v1/files/confirm`

Common causes:
- Missing `locker_id` or `filename` in the request body.
- File does not exist in the locker manifest.

Actions:
- Verify the file was created via `POST /v1/lockers/{locker_id}/files` before confirming.
- Ensure the filename matches exactly (case-sensitive).

### Hash mismatch on confirm

If the gateway computes a SHA-256 hash that differs from what was expected, the upload bytes were corrupted or incomplete.

Actions:
- Re-upload the file bytes to the upload_url.
- Re-confirm after successful upload.

## Fetch-and-store failures

### 422 from `POST /v1/files/fetch-and-store`

Common causes:
- Invalid or unreachable `source_url`.
- SSRF protection blocked the target URL (internal/private IP ranges).

Actions:
- Verify the source_url is a publicly accessible HTTPS URL.
- Do not use localhost, private IPs, or internal hostnames.

### Timeout on fetch-and-store

Large files may exceed the gateway's fetch timeout.

Actions:
- For large files, upload directly via the create_file → PUT upload_url path instead.

## Sandboxed app uploads (ChatGPT/Claude-style runtimes)

In proxied runtime sandboxes, direct local path uploads can fail even when the file exists.

### Deterministic detached upload path (recommended)

Use upload sessions so bytes move out-of-band from the model context:

1) `POST /v1/lockers/{locker_id}/upload-sessions`
- include `receipt_id` and file metadata
- receive `session_id`, `upload_token`, and per-session endpoints

2) `POST /v1/upload-sessions/{session_id}/files/{file_id}/parts`
- send base64 parts using `X-Upload-Token` (or bearer token)
- avoid direct `/mnt/data` path assumptions in sandbox tools

3) `POST /v1/upload-sessions/{session_id}/finalize`
- assembles bytes, confirms upload, runs integrity checks
- returns `viewer_url` + UI button contract

4) `GET /v1/upload-sessions/{session_id}`
- poll terminal status (`complete|partial|failed`)

### Structured recovery fields

Upload-related errors include:
- `details.layer` (`app_orchestrator|sandbox_runtime|mcp|gateway|provider|viewer`)
- `details.recovery_hint`
- `details.next_best_method`

Use these fields directly for automatic fallback decisions.

### Storage purchase guardrail

If a valid `receipt_id` already exists in context, reuse it.
Do not call storage purchase endpoints again unless the user explicitly requests a new purchase.

## Short URL failures (upload_url / download_url)

All upload_url and download_url values are short URLs: `https://api.nukez.xyz/f/{token}`. The gateway decodes the Ed25519-signed token and either proxies or 307-redirects to the underlying storage provider. This redirect is transparent to standard HTTP clients.

### 403 from a short URL

The URL token has expired (default TTL: 30 minutes) or the signature is invalid.

Actions:
- Refresh by calling `GET /v1/lockers/{locker_id}/files/{filename}` (or re-running `create_file`).
- Then use the new `upload_url`/`download_url`.

### 404 from a short URL

The file does not exist in the locker, or the URL token references a file that was deleted.

Actions:
- Call `GET /v1/lockers/{locker_id}/files` (list_files) to see what exists.
- Re-create the file with `POST /v1/lockers/{locker_id}/files`.

### Upload fails with 403 due to Content-Type mismatch

If the API created the file with a locked content type, your PUT must match.

Actions:
- Use the `content_type` you provided at create-time.
- If the API returned `upload_headers`, include them exactly.
- Note: the 307 redirect preserves your Content-Type header.

### Verifying short URL tokens independently

If you want to verify a short URL token was issued by this Nukez gateway:
1) Fetch the public key: `GET /v1/short-url/verify-key` -> `verify_key_hex`
2) Base64url-decode the token, split payload (all but last 64 bytes) from signature (last 64 bytes)
3) Verify Ed25519 signature using the public key

This requires no shared secret and no trust in the gateway.

## Transient 5xx errors

### 500 `IAM_SIGNBLOB_PERMISSION_DENIED` (or similar)

Meaning: the service can't sign storage URLs due to missing IAM permissions.

Actions:
- This is not an agent-recoverable problem. Report it.

### Other 5xx

Actions:
- Retry with exponential backoff + jitter.
- Prefer retries for idempotent operations (GETs, confirms with the same tx, etc.).
- For non-idempotent operations (e.g., creating new files with server-generated names), retry carefully and de-duplicate by listing files.