# Operating hosts

**Spinbox is experimental: APIs change and data may be lost.** What is built and what proves it is on [Status](https://docs.spinbox.dev/reference/status.md); what Spinbox does not do is on [Limitations](https://docs.spinbox.dev/overview/limitations.md).

> Drain, activate and revoke runners, see what runs where, replace a runner's machine, and add a runner outside the autoscaling group.

Runners are also called **hosts**. [An administrator manages them in **Admin → Runners** or with
`spin admin runner`](https://docs.spinbox.dev/reference/status.md#administration). For how runners join, scale and go away, see
[Runners and hosts](https://docs.spinbox.dev/concepts/runners.md).

## Seeing them

```bash
spin admin runner list
spin admin versions          # the control plane's release, and each runner's and its machine's
```

`spin admin versions` says [what is running where](https://docs.spinbox.dev/reference/status.md#administration). A runner's page in the dashboard shows its
state, its capacity and its workspaces.

## Drain and activate

```bash
spin admin runner drain <runner-id>             # takes no new workspace; the running ones stay
spin admin runner drain <runner-id> --suspend   # and moves the running ones off, with memory
spin admin runner activate <runner-id>
```

Drain a runner that should stop taking work for a while. It stays drained until someone activates
it. If workspaces wait for a host while a runner is up, check that the runner is not drained: the
waiting workspace says *the connected runners take no new workspace*.

A runner whose local disk is more than 70% full is cordoned automatically, and uncordoned when it
falls under 65%. The automatic cordon never clears one a person set.

## Revoke

```bash
spin admin runner revoke <runner-id>
```

Revoke is for a machine that should never connect again, and **cannot be undone**.
[Taking a runner out, and bringing it back](https://docs.spinbox.dev/admin/runners.md) says what each action leaves
behind, what a revoke refuses, and how the capacity comes back.

## Replacing a runner's machine on AWS

A runner of the group that should go for good is given back to it, which terminates the machine
and is asked for one less. It drains as it goes, and one that is the only home of a workspace's
disk is refused, naming them:

```bash
spin admin runner remove <runner-id>
spin admin runner remove <runner-id> --taken
```

`--taken` is for a machine that may be compromised: its credentials are deleted and it is cut off
before its group is asked anything, and every disk whose key it was given is marked for re-keying,
as a revoke does. With `--leave-workspaces` as well, it goes even where workspaces' disks are only
there; each of those runs again only once it is recovered onto another runner.

After a revoke, or when a runner is wedged, terminate its instance without lowering the group's
size. The group starts another, which joins by its role:

```bash
aws autoscaling describe-auto-scaling-instances \
  --query 'AutoScalingInstances[?AutoScalingGroupName==`<name>-runners`].[InstanceId,LifecycleState,HealthStatus]' \
  --output table
aws autoscaling terminate-instance-in-auto-scaling-group \
  --instance-id <i-...> --no-should-decrement-desired-capacity
```

If nothing is waiting for a host, the control plane may already have scaled the group to zero,
and there is nothing to replace.

## Workspaces on a runner that is gone

A workspace whose runner was revoked or is gone stops, in the record, and its owner moves it: see
[A workspace on a runner that is gone](https://docs.spinbox.dev/admin/runners.md#a-workspace-on-a-runner-that-is-gone).
The thresholds that let the control plane move them on its own are in the
[installation settings](https://docs.spinbox.dev/admin/configuration.md).

## A runner outside the group

A machine that is not in the autoscaling group joins with a registration token:

```bash
spin admin token create --description "lab machine" --expires 1h
```

A token is one-time unless `--reusable`, lasts 24 hours unless `--expires` says otherwise (at
most 30 days), and is shown once. `--team` makes a runner that only takes the team's workspaces.
The same flags set what the runner advertises and whether it is preemptible.

Such a machine needs KVM, and must run the control plane's release of Spinbox: a host of any other is
refused, and the refusal says so.
