Run Buildkite agents on exe.dev, a fresh VM per job

How it works#

• my-buildkite-base is an ordinary VM. It never runs jobs; it's what each job's VM is copied from.

• my-buildkite runs bk-dispatcher.py, a short Python program. Every 2 seconds it asks Buildkite which jobs are waiting in its queues, and for each one:

  1. reserves the job, so nothing else takes it;
  2. cp my-buildkite-base my-buildkite-job-<random> --sandbox, sized for the job's queue;
  3. asks Buildkite for a job acquisition token, which can only register an agent for that job and expires in 10 minutes;
  4. starts buildkite-agent on the copy with that token and BUILDKITE_AGENT_ACQUIRE_JOB, so it runs the job and exits.

  When the job is finished, cancelled, or failed, the dispatcher stops the agent and deletes the VM. If a VM can't be made, it fails the job with the reason, rather than leaving it waiting.

• Nothing calls the dispatcher. It only makes outbound requests, so my-buildkite stays private: no webhook, no public port.

• The dispatcher holds no Buildkite or exe.dev key. It reaches both through two integrations attached to it alone, so neither is on any VM, the job VMs included.

Before you start#

• A Buildkite organization. The Free plan is enough. You need to be an administrator of the organization, or a maintainer of its cluster.
• An exe.dev account that can create sandboxes. Personal accounts can. On a Work team, an admin turns them on with ssh exe.dev team settings standalone all-users.
• About 20 minutes.

Prepare the base VM#

ssh exe.dev new --name=my-buildkite-base --tag=buildkite

ssh my-buildkite-base.exe.xyz 'curl -fsSL https://exe.dev/deploy/buildkite/base-setup.sh | bash'

The base already has what the default exe.dev image ships: Docker, Go, Python with uv, git, and build tools. Install whatever else your pipelines need, the way you would on any Ubuntu 24.04 machine:

ssh my-buildkite-base.exe.xyz 'sudo apt-get install -y postgresql-client libvips-dev'

Then warm the caches by building your project there once. Jobs run as the same exedev user, so they find ~/go/pkg/mod, ~/.cache/go-build, ~/.npm, ~/.cache/pip, or ~/.cargo already full. From a checkout on your computer:

rsync -a --exclude=.git ./ my-buildkite-base.exe.xyz:warm/
ssh my-buildkite-base.exe.xyz 'cd warm && go build ./... && go test ./... ; cd && rm -rf warm && sync'

(npm ci && npm test, uv sync && uv run pytest, cargo test: whatever your steps run.) Docker images work the same way: docker pull them on the base, and every job has them. See Caches for git.

Don't put secrets on the base. Everything on it is on every job's VM.

Create the queues and the agent token#

In Buildkite, open Agents and your cluster (a new organization has one, Default cluster).

1. Under Queues, select New Queue, choose Self hosted, and give it the key exe. Make a second one, exe-large, for bigger jobs. Each queue is one size of job VM; you set the sizes in Start the dispatcher.
2. Under Agent Tokens, select New Token, describe it as exe.dev dispatcher, and copy the token (it starts with bkct_).

The dispatcher uses that token to find, reserve, and start jobs in the cluster. It never leaves exe.dev's integration: the job VMs get a one-job token instead.

Create the dispatcher VM and its keys#

On your computer:

ssh exe.dev new --name=my-buildkite

Buildkite. Put the agent token in an HTTP proxy integration that only my-buildkite can use. This reads it without echoing it or saving it in your shell history:

printf 'Agent token: '; read -rs T; echo
ssh exe.dev "integrations add http-proxy --name bk-agent --target https://agent.buildkite.com --header 'Authorization:Token $T' --attach vm:my-buildkite"
unset T

The quotes matter: Buildkite wants Authorization: Token …, and the space in it would otherwise split the header in two.

exe.dev. Make an API key that can copy, list, reach, comment on, and delete VMs tagged buildkite, and nothing else, and hand it to a second integration:

ssh exe.dev ssh-key generate-api-key --label=buildkite-dispatcher --tag=buildkite \
    --cmds=cp,ls,rm,ssh,comment --exp=1y --json | jq -r .token \
  | ssh exe.dev integrations add http-proxy --name bk-exe \
    --target https://exe.dev --bearer - --attach vm:my-buildkite

The copies the key makes are tagged buildkite too, so it can delete them. ssh exe.dev ssh-key remove buildkite-dispatcher revokes it.

Check both from the dispatcher VM:

ssh my-buildkite.exe.xyz
curl -s -X POST https://bk-exe.int.exe.xyz/exec -d ls | jq -r '.vms[].vm_name'
curl -s -X POST https://bk-agent.int.exe.xyz/v3/stacks/register -H 'Content-Type: application/json' -d '{}'

The first prints my-buildkite-base, and nothing that isn't tagged buildkite. The second prints {"message":"Cluster queue not found for key: "}: Buildkite accepted the token and refused the empty request. 401 means the token is wrong; an invalid token error from exe.dev means the second integration got something other than the key.

Warning

Attach these two integrations to my-buildkite only (--attach vm:my-buildkite), never by the buildkite tag: every job VM carries that tag, and any job could then create VMs or take other jobs.

Start the dispatcher#

On the VM (ssh my-buildkite.exe.xyz), fetch the dispatcher and write its settings:

mkdir -p ~/bk
curl -fsSL -o ~/bk/bk-dispatcher.py https://exe.dev/deploy/buildkite/bk-dispatcher.py
(umask 077; cat > ~/bk/env <<'EOF'
AGENT_API=https://bk-agent.int.exe.xyz
EXE_API=https://bk-exe.int.exe.xyz
BASE_VM=my-buildkite-base
QUEUES="exe=2 exe-large=4"
MAX_AGENTS=5
EOF
)

• QUEUES: the queue keys this dispatcher serves, each with the vCPUs of its job VMs. Every vCPU comes with 2 GiB of memory, the most a sandbox has, so exe jobs get 2 vCPU / 4 GiB and exe-large jobs 4 vCPU / 8 GiB.
• MAX_AGENTS: at most this many job VMs at once, all queues together; further jobs wait in their queue. Keep it at 10 or under on Buildkite's Free plan, and under your exe.dev plan's VM limit (50 on Personal) minus the VMs you already have.
• Optional: POLL_SECONDS (2) between checks, START_SECONDS (300) for an agent to pick up its job before the dispatcher fails it, MAX_HOURS (6) before any job VM is deleted, and STACK_KEY (the VM's name), which identifies the dispatcher to Buildkite.

Run it as a service, so it comes back after a crash or a reboot:

sudo tee /etc/systemd/system/bk-dispatcher.service >/dev/null <<'EOF'
[Unit]
Description=Buildkite jobs on exe.dev sandboxes
After=network-online.target
Wants=network-online.target

[Service]
User=exedev
EnvironmentFile=/home/exedev/bk/env
ExecStart=/usr/bin/python3 /home/exedev/bk/bk-dispatcher.py
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now bk-dispatcher
journalctl -u bk-dispatcher -n 5 --no-pager -o cat

The log says stack my-buildkite: queues exe (2 vCPU) exe-large (4 vCPU), copies of my-buildkite-base, up to 5 at once, with no register: or reconcile: errors after it. In Buildkite, the exe queue shows as Connected within a few seconds.

Run a pipeline on it#

In a pipeline's steps, send the jobs you want on exe.dev to a queue:

steps:
  - label: ":go: test"
    command: go test ./...
    agents:
      queue: exe

  - label: ":package: release build"
    command: make release
    agents:
      queue: exe-large

Start a build, and watch the dispatcher on the VM:

journalctl -u bk-dispatcher -f -o cat

22:59:54 started my-buildkite-job-16952394 for exe job 01a1adf0-9d47-430b-bdf6-6f6cc3451c15 in 3.4s
22:59:54 started my-buildkite-job-56a17dd1 for exe-large job 01a1adf0-9d4c-46ed-97fe-492fbae01cee in 3.3s
23:00:02 deleted my-buildkite-job-56a17dd1 (job finished)
23:00:10 deleted my-buildkite-job-16952394 (job finished)

In Buildkite, each job names my-buildkite-job-… as its agent. A step with parallelism: 8 gets eight VMs at once, or MAX_AGENTS of them with the rest started as the first ones finish. Cancel a build and its VMs are deleted within seconds.

Moving one step at a time is fine: steps on other queues (Buildkite's hosted agents, your other agents) keep running where they did.

Caches#

The base holds whatever was on its disk when a job's VM was copied: toolchains, dependency caches, build caches, Docker images. Nothing a job writes comes back to it. It costs no time per job, but it's only as fresh as your last warm-up, so re-run the warm-up when dependencies change a lot, or weekly.

In our test, building a Go project of about 150 modules took 44 seconds on a fresh base. After one go build on the base, it took 4.7 seconds, and the whole job, checkout included, went from 48 seconds to 12.

Git. base-setup.sh turns on the agent's git mirrors, so a mirror on the base makes every checkout a short fetch. Name it after the repository URL exactly as your pipeline has it, with every character that isn't a letter or digit replaced by -. On your computer, for a public repository:

repo=https://github.com/you/app.git
ssh my-buildkite-base.exe.xyz "git clone --mirror $repo ~/.buildkite-agent/git-mirrors/$(printf %s "$repo" | sed 's/[^[:alnum:]]/-/g') && sync"

For a private one, clone it with a credential that isn't saved in the mirror, such as git -c http.extraHeader="Authorization: Basic …" clone --mirror …, and let jobs fetch with their own (see Checkout and secrets).

To update the base, change it (ssh my-buildkite-base.exe.xyz) and end with sync: new job VMs are copies of its disk, and anything not yet written to disk isn't in them. Jobs already running keep their copy.

Checkout and secrets#

Job VMs hold no credentials, so give jobs theirs from Buildkite:

• Private GitHub repositories: on each queue's Settings tab, turn on Use GitHub App token for checkout. Each job then clones with a short-lived, read-only token. It needs Buildkite's full-access GitHub App on your GitHub account.
• Other secrets, and SSH keys for other Git hosts: Buildkite secrets, which the agent fetches for the job that asks.

exe.dev can keep some keys off job VMs altogether. An integration attached by the buildkite tag is usable from every job VM, without a key on it: an HTTP proxy to a private package registry or API, or the LLM gateway for tests that call a model. A step calls https://<name>.int.exe.xyz/ and exe.dev adds the credential. Remember that any job can use it.

Just one VM#

For a small project whose builds all come from you, one VM with always-on agents may be all you need: no base, no dispatcher, no integrations, and nothing to pay beyond your plan. Its agents run job after job on the same VM, so each job sees what earlier ones left behind, and any job can read the agent token in the agent's configuration. Keep it to your own code.

ssh exe.dev new --name=my-buildkite-agent
ssh my-buildkite-agent.exe.xyz

curl -fsSL https://keys.openpgp.org/vks/v1/by-fingerprint/32A37959C2FA5C3C99EFBC32A79206696452D198 \
  | sudo gpg --dearmor -o /usr/share/keyrings/buildkite-agent-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/buildkite-agent-archive-keyring.gpg] https://apt.buildkite.com/buildkite-agent stable main" \
  | sudo tee /etc/apt/sources.list.d/buildkite-agent.list
sudo apt-get update && sudo apt-get install -y buildkite-agent
printf 'Agent token: '; read -rs T; echo
sudo sed -i "s/^token=.*/token=\"$T\"/" /etc/buildkite-agent/buildkite-agent.cfg; unset T
printf 'tags="queue=exe"\nspawn=2\n' | sudo tee -a /etc/buildkite-agent/buildkite-agent.cfg
sudo usermod -aG docker buildkite-agent
sudo systemctl enable --now docker buildkite-agent
sudo journalctl -u buildkite-agent -n 5 --no-pager

If the VM restarts without shutting down cleanly, Buildkite can give a job to an agent that no longer exists. The job waits until Buildkite marks that agent lost, about four minutes, and then runs again.

What it costs#

Buildkite charges for its plan the same way whoever runs the agents: Free for up to 5 users, Pro $30 per active user a month. On self-hosted agents, Free runs up to 10 jobs at once; Pro includes 10 connected agents, then $3.50 a month for each. Every job VM is a connected agent while it runs, so MAX_AGENTS is also your agent count.

exe.dev bills the sandboxes by the second, from the copy to the delete: $0.105 an hour per 2 vCPU / 4 GiB, so $0.00175 a minute for 2 vCPU, $0.0035 for 4, $0.007 for 8. Each job VM lives about 10 to 15 seconds longer than its job. my-buildkite and my-buildkite-base are ordinary VMs that share your plan's capacity with your other VMs and cost nothing extra (on Personal, plans start at $15 a month).

Buildkite's hosted agents are $0.004 per vCPU-minute on Linux, also by the second: $0.008 a minute for Small (2 vCPU, 4 GB), $0.016 for Medium (4 vCPU, 16 GB), $0.032 for Large (8 vCPU, 32 GB). Free includes 2,000 vCPU-minutes a month, Pro 4,000.

Some months, on 30 days:

• A small team. 100 jobs a day, 5 minutes each, on 2 vCPU: 15,000 minutes. Buildkite Small: $120, or $112 after Free's 2,000 vCPU-minutes. exe.dev: 3,000 VMs × 5¼ minutes ≈ 263 hours × $0.105 ≈ $27.60. If the warm base brings the jobs to 3 minutes, ≈ $17.10.
• Big builds. 2,000 builds a month, 12 minutes each, on 8 vCPU. Buildkite Large: 24,000 minutes × $0.032 = $768. exe.dev, 8 vCPU / 16 GiB (4 units, $0.42 an hour): ≈ 408 hours ≈ $171.50.

Mind the memory: Buildkite's Medium and Large have 4 GB per vCPU, and a sandbox at most 2 GiB, so a job that needs 16 GB takes 8 vCPU here. Buildkite's hosted agents stay the better deal for builds that fit in the included minutes, and for macOS, Windows, and arm64, which exe.dev doesn't run.

Update#

• The agent. Buildkite releases one every week or two. Re-run the install on the base: it upgrades the package and keeps your configuration. ssh my-buildkite-base.exe.xyz 'curl -fsSL https://exe.dev/deploy/buildkite/base-setup.sh | bash'.
• The dispatcher. Fetch bk-dispatcher.py again as in Start the dispatcher, then sudo systemctl restart bk-dispatcher. Running jobs are unaffected.
• The keys. The exe.dev key above expires in a year; Buildkite agent tokens made in its interface don't expire. To replace either, make a new one, ssh exe.dev integrations remove bk-agent (or bk-exe), and add it again as above.

Troubleshooting#

Start with the dispatcher's log on my-buildkite: journalctl -u bk-dispatcher -n 50 --no-pager -o cat.

• A job waits and the log shows nothing. Its queue isn't in QUEUES, or MAX_AGENTS job VMs are already running, or the queue is paused in Buildkite (the dispatcher respects that).
• register: or reconcile: with HTTP Error 401. The agent token in bk-agent is wrong or revoked. 404 Cluster queue not found means the first key in QUEUES isn't a queue in the token's cluster.
• launch … failed: HTTP Error 422 and the job fails with exe.dev: could not start …. exe.dev refused the copy, and the log says why: vm "…" not found is a BASE_VM that doesn't exist or isn't tagged buildkite; otherwise sandboxes aren't enabled for your team, or the VM limit is reached.
• A job fails with the agent on … did not pick up the job. The agent on the copy didn't start. Install the base again, then look at a job VM while it's up: ssh my-buildkite-job-… journalctl -u buildkite-job.
• A job can't find a tool or a cached package. It wasn't on the base's disk when the job's VM was copied. Install it, sync, and re-run.
• Leftover job VMs. The dispatcher deletes them while it runs. To see them yourself: ssh exe.dev ls --json | jq -r '.vms[].vm_name | select(startswith("my-buildkite-job-"))'.