# Runners and hosts

**Spinbox is experimental: APIs change and data may be lost.** What is built and what proves it is on [Status](https://docs.spinbox.dev/reference/status.md); what Spinbox does not do is on [Limitations](https://docs.spinbox.dev/overview/limitations.md).

> What a runner is, how it joins, how workspaces are placed on it, how runners scale from zero, and what happens when one is lost.

A **runner** is a machine that runs workspaces. In the dashboard and the API a runner is also
called a **host**. On AWS, runners are an autoscaling group of large EC2 machines with local
NVMe disks and nested virtualization, spot by default.

## What a runner does

- Keeps a connection open to the control plane, and runs what the control plane says should
  run on it.
- Starts each workspace as a QEMU/KVM virtual machine, confined to its own user, cgroup and
  empty network namespace.
- Keeps each workspace's disk on its local NVMe, and uploads its sealed layers to the bucket.
- Enforces each workspace's network policies, in nftables, outside the machine.
- Opens sessions to the relay when someone connects to a workspace.

A runner opens every connection it has. It has no inbound rule at all.

## Joining

A runner in the installation's autoscaling group joins **by who it is**. It signs an AWS
`GetCallerIdentity` request with its instance role, bound to this installation and to its own
id, and sends it to the control plane. The control plane passes it to AWS STS and checks that
the answer names the runners' role. There is no token to create, store or rotate.

A machine outside the group can join with a one-time registration token from Admin → Runners.

A runner must run the control plane's release. One of any other, older or newer, is refused when it
registers, and the refusal says which release to run.

## Placement

The control plane places a workspace on an **active** runner with room for its disk and
memory. It prefers the runner holding fewer of the same team's workspaces, so one team does not
pile onto one machine. A runner whose local disk is more than 70% full stops taking new
workspaces until it is back under 65%.

A preset may name a **host class**, a kind of runner, and its workspaces are placed only on
runners of that class. An installation the module's current release makes has one class,
`default`, and every runner is of it. A spot class can be set to fall back: when AWS refuses spot
capacity for its group, the control plane runs the group on demand for 30 minutes, then tries spot
again.

## Scaling from zero

On AWS [the control plane sizes the runners' group itself](https://docs.spinbox.dev/reference/status.md#administration). It knows what nothing else does: that
a workspace is waiting, and when the last one stopped.

- With no runner up, the first workspace to start waits in *Waiting for a host* while the control
  plane asks for one. It is placed on the runner as soon as it registers.
- If a workspace has waited ten minutes, and no runner was added in that time, the control plane
  adds one.
- When nothing has run for [`runner_idle_minutes`](https://docs.spinbox.dev/reference/terraform/root.md#runner_idle_minutes), it scales the group back to
  none. During the installation's `quiet_hours`, it does so as soon as nothing is running.
- It never asks for more than `max_runners`.

While workspaces run and none waits, a runner that holds nothing only it has, for as long as
`runner_idle_minutes`, is emptied and given back to the group, one at a time.

## When a runner goes away

A runner that is going away empties itself first. It suspends each of its workspaces, with
memory, to the bucket, and each one resumes on another runner from where it was. This happens
when:

- [AWS reclaims a spot runner](https://docs.spinbox.dev/reference/status.md#administration) (about two minutes' notice),
- the autoscaling group removes a runner (the group holds it for `drain_seconds`),
- a release is rolled out and runners are replaced.

A runner restarting its own process is not a drain: the virtual machines keep running, and the
new process adopts them.

## Operating runners

| Action | The runner | Its workspaces | Undo |
|---|---|---|---|
| **Drain** | Stays connected, takes no new workspace | Keep running; with `--suspend`, are moved off | Activate |
| **Activate** | Takes workspaces again | — | — |
| **Revoke** | Its credentials are deleted and it is disconnected for good | Refused while it holds any, unless you choose to leave them | None: replace the machine |

See [Operating hosts](https://docs.spinbox.dev/admin/hosts.md).

## When a runner is lost

A runner can vanish without draining: the machine loses power, or its network. The control plane
does not guess. Its workspaces stay where they are, and each one has lost only what the runner
had not yet uploaded.

- The owner moves one with `spin workspace recover <workspace>`, which says how much would be
  lost and waits for `--accept-data-loss`.
- An administrator can let the control plane move workspaces on its own, after a runner has
  been silent for `rehome_after_seconds`, when no more than `rehome_max_unpublished_bytes` would
  be lost: [the workspace is moved, and its owner is told when it waits for them instead](https://docs.spinbox.dev/reference/status.md#administration).
  This is off by default.

[The owner is told by email](https://docs.spinbox.dev/reference/status.md#observability):
when a workspace is held after its host was lost, when a stop could not be taken, when it was
stopped for not responding, and when it was moved off a lost host. Each event can be switched
off.
