Skip to main content

​
Storage migration: copy, verify, cut over, rollback

This runbook migrates CaseBender objects without changing object keys. It uses copy-first semantics: the source remains authoritative and intact until the rollback window expires. It applies to local filesystems, MinIO/S3-compatible storage, AWS S3, and GCS through rclone remotes.

​
Preconditions

  • Read the storage support policy and release notes for both providers.
  • Fix or account for the legacy MinIO-only attachment deletion path before moving to a non-MinIO destination.
  • Confirm destination capacity, encryption, versioning, retention, lifecycle, object-size, metadata, and naming behavior.
  • Create least-privilege source-read and destination-write migration identities.
  • Configure TLS trust; never use --no-check-certificate.
  • Take and test a consistency backup of PostgreSQL and source storage.
  • Record object count, total bytes, source versions/snapshots, and configuration.
  • Set a change window that permits write quiescence and rollback.
Examples below use source:casebender and destination:casebender. Keep rclone configuration and logs outside the repository and protect them as sensitive.

​
1. Inventory and dry run

Review unsupported metadata warnings. Provider-specific encryption, retention, legal hold, ACL, and version history may not copy as ordinary object metadata; configure those controls at the destination and preserve source versions in the backup. Never use sync for the initial copy because it can delete destination objects.

​
2. Seed copy while the application is online

Tune concurrency below provider throttle limits. Retry failed objects and retain the complete log. Do not infer integrity from ETags: multipart and encrypted objects may have non-MD5 ETags.

​
3. Quiesce writes and capture the consistency point

Block user and integration writes using the release’s maintenance procedure. Pause ingestion and workers only after the queue is drained or durably retained. Record the database timestamp/LSN, source bucket version/snapshot, and deployment replicas. Verify no storage writes are occurring. Run the final delta:
Do not delete or disable the source.

​
4. Verify before cutover

rclone check --download hashes downloaded content and avoids provider ETag ambiguity. Require zero missing, changed, or unreadable objects. Investigate count differences from provider marker objects or local .meta.json sidecars; do not waive differences without a recorded explanation. Run the destination contract test from the application network:
For GCS or local examples, see the script usage. Also sample high-value evidence, large multipart objects, Unicode names, empty files, MIME metadata, and retained objects.

​
5. Cut over

  1. Save the old provider configuration and Secret resource version in the encrypted change record.
  2. Change only the explicit provider variables and credentials. Preserve bucket contents and object keys.
  3. Restart the web application and wait for readiness.
  4. Run application upload/download/list/copy/delete tests and verify SHA-256.
  5. Verify existing attachments and evidence across several ages and sizes.
  6. Resume workers and ingestion, then user writes.
  7. Monitor storage errors, failed deletes, latency, throttling, queue depth, and audit events continuously through the rollback window.
Do not run database key rewrites unless a release-specific migration explicitly requires them.

​
6. Rollback

Rollback is safe only while the old source is retained and new writes can be reconciled.
  1. Re-enter maintenance mode and quiesce writes.
  2. Record all objects written to the destination since cutover.
  3. Copy the reverse delta to the source without deletion:
  1. Require a clean check, then restore the prior provider configuration and credential version.
  2. Restart, run application lifecycle/integrity tests, and resume traffic.
  3. Keep both stores and all evidence until incident/change review completes.
If source retention or policy prevents the reverse copy, stop and restore the recorded consistency backup; do not improvise destructive synchronization.

​
7. Closeout

  • Reconcile final counts/bytes and archive hashes, logs, tool versions, approvals, configuration versions, and sampled application results.
  • Rotate temporary migration credentials.
  • Keep source read-only for the approved rollback period.
  • After formal sign-off and legal/retention review, remove source data using the provider’s audited disposal process.
  • Update the support record with provider product/version, TLS/CA details, contract result, performance result, backup result, and rollback exercise.