# Runners: taking one out, and bringing it back

**Spinbox is experimental: APIs change and data may be lost.** What is built and what proves it is on [Status](https://docs.spinbox.dev/reference/status.md); what Spinbox does not do is on [Limitations](https://docs.spinbox.dev/overview/limitations.md).

> What draining, cordoning, activating and revoking a runner do, what each leaves behind, and how to bring the capacity back.

<!-- Generated by tools/refgen from spin:docs/runners.md. Edit the source, not this file. -->

What an administrator does to a runner from Admin → Runners (or `spin admin runner`), what
each action leaves behind, and how to undo it. The states are
`internal/lifecycle`'s; placement leaves out every host that is not `ACTIVE` (`refusingPlacement`
in `internal/controlplane/server/workspaceserver`).

| Action | What happens to the host | Its workspaces | Undo |
|---|---|---|---|
| **Drain** | `DRAINING`: connected, takes no new workspace | keep running | **Activate** |
| **Cordon** (an update with `--evacuate`, a revoke) | `CORDONED`: the same, by another hand | keep running | **Activate** |
| **Activate** | `ACTIVE`: placement may choose it again | - | - |
| **Revoke** | credentials deleted, stream cut, `CORDONED`; its workspaces' supervisor credentials revoked | refused while any is on it, unless left on purpose | none: a new machine |

Drain is what you want when a runner should stop taking work for a while. Revoke is for a machine
that should never connect again - it is being replaced, or it may be compromised: it cannot come
back as the same host, and **Activate does not undo it**.

## Nothing starts, and a runner is up

The control plane says, for every placement it could not make:

```
WRN SelectHost: no suitable host found connected_hosts=1 outcome=host_takes_no_work
```

and a waiting workspace says "the connected runners take no new workspace". Every connected host
is drained or cordoned. List them and activate the one that should take work:

```bash
spin admin runner list
spin admin runner activate <runner-id>
```

or **Activate runner** in the runner's menu. The next placement chooses it.

## Revoking a runner

A runner that holds any workspace's disk - running, stopped or failed - is refused, and the
refusal names them: revoked, each could run again only by being moved to another runner, giving up
what this one had not published. Delete or move them first, or revoke leaving them:

```bash
spin admin runner revoke <runner-id> --leave-workspaces
```

Once revoked, the control plane refuses everything the runner sends, and the runner does not stop
trying:

```
WRN Invalid or expired runner token host_id=<host-id>
WRN Request failed ... path=/spin.agent.v1.VolumeService/Heartbeat status=401
```

The host id is burned: its row stays `CORDONED` and disconnected, and nothing placed will reach it.
Bring the capacity back by giving the installation a new machine, which joins as a new host:

- **On AWS**, replace the instance in the runners' group. spin-stack/spin-terraform-aws's README
  has the command, under *Operating it*. The group starts another, which joins by its role.
- **Self-hosted**, boot the machine again with a new registration token
  (`spin admin token create`) where its document's `install.join.token_at` names it: spin-boot's
  join spends it and the host registers again.

If the machine may be compromised, do not reuse its disk.

## A workspace on a runner that is gone

A workspace whose runner was revoked or declared dead stops at once - in the record, since nothing
runs there any more - and a start of it says to move it. Once a live runner is up, **Move to
another runner** on the workspace's page, or:

```bash
spin workspace recover <workspace>
spin workspace recover <workspace> --accept-data-loss
```

The first changes nothing and says how much its old runner had not published; the second moves it
there, and the workspace is stopped, to be started when its owner is ready. A runner that is only
restarting is waited for instead, and the workspace comes back with nothing lost.

Setting the installation's rehome threshold (`rehome_after_seconds`,
`rehome_max_unpublished_bytes`) makes the control plane move them itself, under that cost.
