# Remove

**Spinbox is experimental: APIs change and data may be lost.** What is built and what proves it is on [Status](https://docs.spinbox.dev/reference/status.md); what Spinbox does not do is on [Limitations](https://docs.spinbox.dev/overview/limitations.md).

> Delete an installation on purpose: what refuses to be destroyed, why, and the steps to remove it anyway.

<!-- Generated by tools/refgen from spin-terraform-aws's README.md. Edit the README, not this file. -->

An installation keeps its data past a plain `tofu destroy`: the volumes, the database's archive
and the logs are in buckets a destroy does not empty, and every bucket refuses object writes and
deletes from outside the VPC - the root account's included, so they cannot be emptied from a
laptop either. Removing one is said first, applied, and only then destroyed:

1. Set `decommission = true` on the module and apply. Nothing but the buckets changes: their
   VPC-only policies go, and each is marked to be emptied by a destroy - every version, the
   GOVERNANCE retention bypassed.

   ```bash
   tofu apply   # with decommission = true in module "spin"
   ```

2. Lift every legal hold. spin holds what each image is rebuilt from, and the provider's destroy
   (hashicorp/aws 6.66) lifts a hold and then deletes that version without the bypass, which the
   retention refuses: the destroy stops on the volumes' bucket.

   ```bash
   for b in $(tofu show -json | jq -r '.. | objects | select(.mode? == "managed" and .type? == "aws_s3_bucket" and .values.object_lock_enabled? == true) | .values.bucket'); do
     aws s3api list-object-versions --bucket $b --output json | jq -r '(.Versions // [])[] | [.Key, .VersionId] | @tsv' |
       while IFS=$'\t' read -r key id; do aws s3api put-object-legal-hold --bucket $b --key "$key" --version-id "$id" --legal-hold Status=OFF; done
   done
   ```

3. The KMS keys - the encryption key's, the volumes' and the identity's - and the CA refuse to be
   destroyed, because an apply that replaced one would leave an installation nothing can open.
   Note the keys, take them out of the state, destroy, and schedule each key's deletion (the
   destroy takes their aliases):

   ```bash
   keys=$(tofu show -json | jq -r '.. | objects | select(.mode? == "managed" and .type? == "aws_kms_key") | .values.key_id')
   tofu state rm $(tofu state list | grep -E '\.aws_kms_key\.|\.tls_(private_key|self_signed_cert)\.ca$')
   tofu destroy
   for id in $keys; do aws kms schedule-key-deletion --key-id $id --pending-window-in-days 7; done
   ```

The apply has to come first: a destroy empties a bucket as its state says, so `decommission`
set only on the destroy changes nothing - it stops with `BucketNotEmpty`. A destroy that already
stopped takes the same apply, limited to the buckets and policies the state still has: a bucket
named that the destroy already removed would be made again. Its plan is those buckets'
`force_destroy` turned on and their policies removed, and nothing else; then step 2 again:

```bash
tofu apply $(tofu state list | grep -E '\.aws_s3_bucket(_policy)?\.[a-z]+(\[0\])?$' | sed 's/^/-target=/')
# step 2's loop
tofu destroy
```

What is left are the keys, scheduled for deletion in seven days. The state bucket and its key
(`bootstrap/`) are every installation's in the account and region, and are removed apart, after
the last of them. `.github/workflows/fresh-install.yml` in spin-terraform-aws runs all of it every
week - bootstrap, an installation, its first sign-in and this removal - in an account of its own,
and says what that account needs.
