> ## Documentation Index
> Fetch the complete documentation index at: https://docs.casebender.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage Health and Troubleshooting

> Diagnose readiness, scanner, CA, permission, canary, and dead-letter failures

Use health responses for routing and sanitized triage. Use protected logs,
metrics, provider audit records, and database state for diagnosis.

## Web health endpoints

| Endpoint            | Meaning                                                     | Success                | Failure                                             |
| ------------------- | ----------------------------------------------------------- | ---------------------- | --------------------------------------------------- |
| `/api/health/live`  | Web process is running; no external checks                  | `200 {"status":"ok"}`  | Process/network failure                             |
| `/api/health/ready` | Storage profiles are reachable and the deep canary is fresh | `200`, `status: ready` | `503`, `status: not_ready` and sanitized `category` |

Readiness categories are only `configuration`, `authentication`, `tls`,
`storage`, `capability`, and `canary_stale`. The response intentionally omits
providers, endpoints, bucket names, object keys, credentials, and raw errors.

```bash theme={null}
curl --fail --silent --show-error \
  'https://<casebender-host>/api/health/live'
curl --fail --silent --show-error \
  'https://<casebender-host>/api/health/ready'
```

Do not use liveness to decide that writes are safe. Do not put a credential in
a probe URL.

## Deep canary

The scheduled deep canary writes random bytes to `ephemeral`, records SHA-256
metadata, reads and hashes the exact bytes, copies and verifies them, then
deletes both objects. Readiness reports `canary_stale` when no successful
canary exists within the freshness window.

For a stale canary:

1. confirm web instrumentation started the canary scheduler;
2. inspect `storage.readiness`, `storage.operation.*`, and provider latency;
3. confirm `ephemeral` permits conditional create, read, copy, and delete;
4. check worker/web clock and event-loop saturation;
5. verify lifecycle policy is not deleting canary objects during the
   transaction; and
6. run the provider contract from the same network and identity context.

Do not permanently lengthen the age threshold to hide failures.

## Category triage

### `configuration`

Validate `STORAGE_CONFIG_FILE` readability/mode and strict JSON; all three
profiles must exist. Use canonical provider IDs `s3`, `gcs`, `azure`, or
`local`. Production cannot use `local`.

### `authentication`

Check workload-identity binding, role scope, token audience, credential
rotation overlap, and provider audit denials. Web and worker need matching
access. Do not print tokens, run `env`, or copy Secret data into a ticket.

### `tls`

Check endpoint DNS/SAN, private CA mount, complete issuing chain, expiry,
proxy trust, and pod restart after CA rotation. Keep verification enabled;
never use HTTP, `--insecure`, or `rejectUnauthorized: false`.

### `storage`

Confirm the existing bucket/container, route, DNS, egress, quota, throttling,
capacity, and required object operations. Health checks do not create missing
storage.

### `capability`

When WORM is required, verify profile `requireWorm`, versioning, object
retention/immutability, and legal hold. Re-run live qualification after policy
changes.

## Scanner outage

Symptoms include increasing `storage.quarantine.depth`,
`storage.quarantine.oldest_age_seconds`, `storage.scanner.failure`, outbox
retries, and eventual dead letters.

Check the `clamd` socket/host, TLS CA, server name, size limit, timeout, network
policy, engine health, and signature update status. Restore the scanner, then
allow normal idempotent retries or use the approved replay procedure. Objects
must remain quarantined until a terminal clean verdict; never mark them clean
manually or bypass scanning.

Cloud Run reference deploys set `MALWARE_SCANNER_PROVIDER=skip` because there
is no ClamAV sidecar. That is an explicit integrity-only exception. If
attachments stay on HTTP 409, confirm the worker started the storage mutation
processor and is not missing that variable.

## Mutation dead letters

Monitor `storage.outbox.deadletter`, `storage.outbox.pending`, and
`storage.outbox.deadlettered`. Inspect protected
`StorageMutationOutbox` records by ID/status/operation without exporting
payloads or object keys unnecessarily.

1. classify scanner, permission, retention, TLS, missing-object, or provider
   failure;
2. repair the cause;
3. confirm the object still matches its stored SHA-256 and exact version;
4. replay through the approved idempotent queue operation; and
5. confirm completion and audit evidence.

Never delete a dead-letter row to make a dashboard green. Retention-blocked
deletes are policy failures to resolve, not objects to force-delete.

## Migration and reconciliation

Monitor `storage.migration.result`,
`storage.reconciliation.missing`,
`storage.reconciliation.orphan_observed`,
`storage.reconciliation.orphan_confirmed`, and reconciliation outbox repair
metrics.

* Missing expected objects are moved back to a quarantined/failed-integrity
  state.
* Orphans are observed first and confirmed only after the grace period.
* Migration retries can reach `DEAD_LETTER`; keep the source and investigate
  before replay.

Do not automatically delete confirmed orphans. Correlate them with upload
sessions, migration ledgers, provider versions, retention, and legal holds.

## Safe validation

Use a dedicated prefix and workload-equivalent identity:

```bash theme={null}
STORAGE_TEST_PROVIDER=s3 \
STORAGE_TEST_BUCKET='<dedicated-test-bucket>' \
AWS_REGION='<region>' \
./scripts/storage/validate-storage.sh
```

For ODF/Ceph, use `./scripts/storage/validate-ceph-rgw.sh` with the exact
version and private CA. Sanitize outputs before attaching them to an incident.