Skip to main content
Use health responses for routing and sanitized triage. Use protected logs, metrics, provider audit records, and database state for diagnosis.

​
Web health endpoints

Readiness categories are only configuration, authentication, tls, storage, capability, and canary_stale. The response intentionally omits providers, endpoints, bucket names, object keys, credentials, and raw errors.
Do not use liveness to decide that writes are safe. Do not put a credential in a probe URL.

​
Deep canary

The scheduled deep canary writes random bytes to ephemeral, records SHA-256 metadata, reads and hashes the exact bytes, copies and verifies them, then deletes both objects. Readiness reports canary_stale when no successful canary exists within the freshness window. For a stale canary:
  1. confirm web instrumentation started the canary scheduler;
  2. inspect storage.readiness, storage.operation.*, and provider latency;
  3. confirm ephemeral permits conditional create, read, copy, and delete;
  4. check worker/web clock and event-loop saturation;
  5. verify lifecycle policy is not deleting canary objects during the transaction; and
  6. run the provider contract from the same network and identity context.
Do not permanently lengthen the age threshold to hide failures.

​
Category triage

​
configuration

Validate STORAGE_CONFIG_FILE readability/mode and strict JSON; all three profiles must exist. Use canonical provider IDs s3, gcs, azure, or local. Production cannot use local.

​
authentication

Check workload-identity binding, role scope, token audience, credential rotation overlap, and provider audit denials. Web and worker need matching access. Do not print tokens, run env, or copy Secret data into a ticket.

​
tls

Check endpoint DNS/SAN, private CA mount, complete issuing chain, expiry, proxy trust, and pod restart after CA rotation. Keep verification enabled; never use HTTP, --insecure, or rejectUnauthorized: false.

​
storage

Confirm the existing bucket/container, route, DNS, egress, quota, throttling, capacity, and required object operations. Health checks do not create missing storage.

​
capability

When WORM is required, verify profile requireWorm, versioning, object retention/immutability, and legal hold. Re-run live qualification after policy changes.

​
Scanner outage

Symptoms include increasing storage.quarantine.depth, storage.quarantine.oldest_age_seconds, storage.scanner.failure, outbox retries, and eventual dead letters. Check the clamd socket/host, TLS CA, server name, size limit, timeout, network policy, engine health, and signature update status. Restore the scanner, then allow normal idempotent retries or use the approved replay procedure. Objects must remain quarantined until a terminal clean verdict; never mark them clean manually or bypass scanning. Cloud Run reference deploys set MALWARE_SCANNER_PROVIDER=skip because there is no ClamAV sidecar. That is an explicit integrity-only exception. If attachments stay on HTTP 409, confirm the worker started the storage mutation processor and is not missing that variable.

​
Mutation dead letters

Monitor storage.outbox.deadletter, storage.outbox.pending, and storage.outbox.deadlettered. Inspect protected StorageMutationOutbox records by ID/status/operation without exporting payloads or object keys unnecessarily.
  1. classify scanner, permission, retention, TLS, missing-object, or provider failure;
  2. repair the cause;
  3. confirm the object still matches its stored SHA-256 and exact version;
  4. replay through the approved idempotent queue operation; and
  5. confirm completion and audit evidence.
Never delete a dead-letter row to make a dashboard green. Retention-blocked deletes are policy failures to resolve, not objects to force-delete.

​
Migration and reconciliation

Monitor storage.migration.result, storage.reconciliation.missing, storage.reconciliation.orphan_observed, storage.reconciliation.orphan_confirmed, and reconciliation outbox repair metrics.
  • Missing expected objects are moved back to a quarantined/failed-integrity state.
  • Orphans are observed first and confirmed only after the grace period.
  • Migration retries can reach DEAD_LETTER; keep the source and investigate before replay.
Do not automatically delete confirmed orphans. Correlate them with upload sessions, migration ledgers, provider versions, retention, and legal holds.

​
Safe validation

Use a dedicated prefix and workload-equivalent identity:
For ODF/Ceph, use ./scripts/storage/validate-ceph-rgw.sh with the exact version and private CA. Sanitize outputs before attaching them to an incident.