Skip to main content
This runbook migrates CaseBender objects without changing object keys. It uses copy-first semantics: the source remains authoritative and intact until the rollback window expires. It applies to local filesystems, MinIO/S3-compatible storage, AWS S3, and GCS through rclone remotes.

​
Preconditions

  • Read the storage support policy and release notes for both providers.
  • Confirm the release uses the provider-neutral durable deletion outbox. The historical MinIO-only deletion gap is fixed; do not carry that stale limitation into a new design.
  • Confirm destination capacity, encryption, versioning, retention, lifecycle, object-size, metadata, and naming behavior.
  • Create least-privilege source-read and destination-write migration identities.
  • Configure TLS trust; never use --no-check-certificate.
  • Take and test a consistency backup of PostgreSQL and source storage.
  • Record object count, total bytes, source versions/snapshots, and configuration.
  • Set a change window that permits write quiescence and rollback.
Generate and protect the authoritative PostgreSQL object inventory:
Every migrated item must identify an exact source object version/generation, size, and SHA-256. PostgreSQL and those exact versions are one consistency set. Do not migrate a floating “latest” object when a version is recorded. Examples below use source:casebender and destination:casebender. Keep rclone configuration and logs outside the repository and protect them as sensitive.

​
1. Inventory and dry run

Review unsupported metadata warnings. Provider-specific encryption, retention, legal hold, ACL, and version history may not copy as ordinary object metadata; configure those controls at the destination and preserve source versions in the backup. Never use sync for the initial copy because it can delete destination objects.

​
2. Seed copy while the application is online

Tune concurrency below provider throttle limits. Retry failed objects and retain the complete log. Do not infer integrity from ETags: multipart and encrypted objects may have non-MD5 ETags.

​
3. Quiesce writes and capture the consistency point

Block user and integration writes using the release’s maintenance procedure. Pause ingestion and workers only after the queue is drained or durably retained. Record the database timestamp/LSN, source bucket version/snapshot, and deployment replicas. Verify no storage writes are occurring. Run the final delta:
Do not delete or disable the source.

​
4. Verify before cutover

rclone check --download hashes downloaded content and avoids provider ETag ambiguity. Require zero missing, changed, or unreadable objects. Investigate count differences from provider marker objects or local .meta.json sidecars; do not waive differences without a recorded explanation. Run the destination contract test from the application network:
For GCS or local examples, see the script usage. Also sample high-value evidence, large multipart objects, Unicode names, empty files, MIME metadata, and retained objects.

​
5. Cut over

  1. Save the old provider configuration and Secret resource version in the encrypted change record.
  2. Confirm the StorageMigrationLedger entries are VERIFIED. The migration processor copies first, reads back, verifies SHA-256, records the destination provider version, and only then transitions to CUTOVER.
  3. Change only the explicit provider profiles and credentials. Preserve bucket contents and durable keys.
  4. Restart the web application and wait for /api/health/ready.
  5. Run application upload/download/list/copy/delete tests and verify SHA-256.
  6. Verify existing attachments and evidence across several ages and sizes.
  7. Resume workers and ingestion, then user writes.
  8. Monitor storage errors, failed deletes, latency, throttling, queue depth, and audit events continuously through the rollback window.
Do not run database key rewrites unless a release-specific migration explicitly requires them. During a documented compatibility window, legacy references may be read from the source while ledger-backed objects use the destination. Do not implement unbounded dual writes. Close legacy reads only after reconciliation confirms there are no missing references and rollback approval permits it.

​
6. Rollback

Rollback is safe only while the old source is retained and new writes can be reconciled.
  1. Re-enter maintenance mode and quiesce writes.
  2. Record all objects written to the destination since cutover.
  3. Copy the reverse delta to the source without deletion:
  1. Require a clean check, then restore the prior provider configuration and credential version.
  2. Restart, run application lifecycle/integrity tests, and resume traffic.
  3. Keep both stores and all evidence until incident/change review completes.
If source retention or policy prevents the reverse copy, stop and restore the recorded consistency backup; do not improvise destructive synchronization.

​
7. Closeout

  • Reconcile final counts/bytes and archive hashes, logs, tool versions, approvals, configuration versions, and sampled application results.
  • Rotate temporary migration credentials.
  • Keep source read-only for the approved rollback period.
  • After formal sign-off and legal/retention review, remove source data using the provider’s audited disposal process.
  • Update the support record with provider product/version, TLS/CA details, contract result, performance result, backup result, and rollback exercise.
Run an isolated restore verification before source disposal:
Record actual recovery point and recovery time. RPO/RTO objectives without a measured restore and rollback exercise are not evidence.

​
Provider-specific caveats

  • S3/Ceph RGW: preserve version IDs and delete markers; ordinary copies may not retain Object Lock/legal-hold state. Requalify the exact product version and private CA.
  • GCS: preserve numeric generations; bucket retention, object retention, event-based holds, and CMEK policy need separate destination attestation.
  • Azure: preserve blob version IDs; immutability and legal hold are destination-specific and copied blobs receive new version IDs.
  • Legacy MinIO: keep both the original volume snapshot and S3 API export. Never import MinIO’s internal filesystem layout into another provider.
See Backup and Restore and MinIO Lifecycle.