> ## Documentation Index
> Fetch the complete documentation index at: https://docs.casebender.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage migration runbook

# Storage migration: copy, verify, cut over, rollback

This runbook migrates CaseBender objects without changing object keys. It uses
copy-first semantics: the source remains authoritative and intact until the
rollback window expires. It applies to local filesystems, MinIO/S3-compatible
storage, AWS S3, and GCS through `rclone` remotes.

## Preconditions

* Read the storage support policy and release notes for both providers.
* Fix or account for the legacy MinIO-only attachment deletion path before
  moving to a non-MinIO destination.
* Confirm destination capacity, encryption, versioning, retention, lifecycle,
  object-size, metadata, and naming behavior.
* Create least-privilege source-read and destination-write migration identities.
* Configure TLS trust; never use `--no-check-certificate`.
* Take and test a consistency backup of PostgreSQL and source storage.
* Record object count, total bytes, source versions/snapshots, and configuration.
* Set a change window that permits write quiescence and rollback.

Examples below use `source:casebender` and `destination:casebender`. Keep rclone
configuration and logs outside the repository and protect them as sensitive.

## 1. Inventory and dry run

```sh theme={null}
rclone version
rclone size source:casebender --json > source-size.before.json
rclone lsf source:casebender --recursive --files-only \
  > source-objects.before.txt
rclone copy source:casebender destination:casebender \
  --checksum --metadata --dry-run --log-level INFO \
  --log-file migration-dry-run.log
```

Review unsupported metadata warnings. Provider-specific encryption, retention,
legal hold, ACL, and version history may not copy as ordinary object metadata;
configure those controls at the destination and preserve source versions in the
backup. Never use `sync` for the initial copy because it can delete destination
objects.

## 2. Seed copy while the application is online

```sh theme={null}
rclone copy source:casebender destination:casebender \
  --checksum --metadata --fast-list --transfers 8 --checkers 16 \
  --log-level INFO --log-file migration-seed.log
```

Tune concurrency below provider throttle limits. Retry failed objects and retain
the complete log. Do not infer integrity from ETags: multipart and encrypted
objects may have non-MD5 ETags.

## 3. Quiesce writes and capture the consistency point

Block user and integration writes using the release's maintenance procedure.
Pause ingestion and workers only after the queue is drained or durably retained.
Record the database timestamp/LSN, source bucket version/snapshot, and deployment
replicas. Verify no storage writes are occurring.

Run the final delta:

```sh theme={null}
rclone copy source:casebender destination:casebender \
  --checksum --metadata --fast-list --transfers 8 --checkers 16 \
  --log-level INFO --log-file migration-final.log
```

Do not delete or disable the source.

## 4. Verify before cutover

```sh theme={null}
rclone check source:casebender destination:casebender \
  --download --one-way --combined migration-check.txt
rclone size source:casebender --json > source-size.final.json
rclone size destination:casebender --json > destination-size.final.json
```

`rclone check --download` hashes downloaded content and avoids provider ETag
ambiguity. Require zero missing, changed, or unreadable objects. Investigate
count differences from provider marker objects or local `.meta.json` sidecars;
do not waive differences without a recorded explanation.

Run the destination contract test from the application network:

```sh theme={null}
STORAGE_TEST_PROVIDER=s3 \
STORAGE_TEST_ENDPOINT=https://storage.example.com \
STORAGE_TEST_BUCKET=casebender \
AWS_REGION=us-east-1 \
./scripts/storage/validate-storage.sh
```

For GCS or local examples, see the script usage. Also sample high-value evidence,
large multipart objects, Unicode names, empty files, MIME metadata, and retained
objects.

## 5. Cut over

1. Save the old provider configuration and Secret resource version in the
   encrypted change record.
2. Change only the explicit provider variables and credentials. Preserve bucket
   contents and object keys.
3. Restart the web application and wait for readiness.
4. Run application upload/download/list/copy/delete tests and verify SHA-256.
5. Verify existing attachments and evidence across several ages and sizes.
6. Resume workers and ingestion, then user writes.
7. Monitor storage errors, failed deletes, latency, throttling, queue depth, and
   audit events continuously through the rollback window.

Do not run database key rewrites unless a release-specific migration explicitly
requires them.

## 6. Rollback

Rollback is safe only while the old source is retained and new writes can be
reconciled.

1. Re-enter maintenance mode and quiesce writes.
2. Record all objects written to the destination since cutover.
3. Copy the reverse delta to the source without deletion:

```sh theme={null}
rclone copy destination:casebender source:casebender \
  --checksum --metadata --fast-list \
  --log-level INFO --log-file rollback-copy.log
rclone check destination:casebender source:casebender \
  --download --one-way --combined rollback-check.txt
```

4. Require a clean check, then restore the prior provider configuration and
   credential version.
5. Restart, run application lifecycle/integrity tests, and resume traffic.
6. Keep both stores and all evidence until incident/change review completes.

If source retention or policy prevents the reverse copy, stop and restore the
recorded consistency backup; do not improvise destructive synchronization.

## 7. Closeout

* Reconcile final counts/bytes and archive hashes, logs, tool versions, approvals,
  configuration versions, and sampled application results.
* Rotate temporary migration credentials.
* Keep source read-only for the approved rollback period.
* After formal sign-off and legal/retention review, remove source data using the
  provider's audited disposal process.
* Update the support record with provider product/version, TLS/CA details,
  contract result, performance result, backup result, and rollback exercise.