# Architecture Tyler Youk · updated 2026-10-31 · https://tyleryouk.com/writing/architecture The fleet has five layers, listed in the order they depend on each other. The **hosts** exist, the **agents** run on them, the **runtime** starts an agent, the **services** stay up between runs, and the **secret store** is what everything authenticates against. ## The vocabulary {#the-vocabulary} The fleet calls its structural choices **architectural patterns**. There are 14 of them, and the fleet chose that term on purpose rather than drifting into it. Everything below is one of two things. * A **component** is something that *runs* or *holds state*, such as the box, the sweeper or the secret store. The fleet keeps a registry of components, one document each, and this page describes that registry. * An **architectural pattern** is one structural choice, written so that a reader on a different base can **adopt it or reject it**. Several patterns do not run at all. Decision discipline and liveness-vs-existence are policies, not processes. That second point is why this site does not call the set *mechanisms*, *components*, *Fleet Architecture* or *Fleet Infrastructure*. Each of those names fits the half that runs and is wrong for the policy half. A name for the set has to pass the adopt-or-reject test, and only *pattern* passes it. Elsewhere on these pages "mechanism" keeps its ordinary English meaning. The rule applies only to the name of the set. The fleet's canonical list of the 14 patterns is in its knowledge base, at `wi/work/reference/architectural-patterns.md`. The pages on this site explain them. Nothing on this page is a health reading. Whether a host is online, whether an agent can run *today*, and how full the disk is all change often, and a written page that stated them would be wrong within days. Where this page needs one of these values, it gives **the exact command and the box that answers it** instead of a number. The fleet's cockpit at utopiamodels.ai shows the current state. ## Hosts {#hosts} ### novaserver, the fleet's box {#novaserver--the-fleets-box} One machine does almost everything. It is an HP EliteDesk 805 G6 running Ubuntu 24.04. You can reach it on its tailnet address (written `100.x.x.x` on this site) and on the LAN. | | | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CPU | AMD Ryzen 5 PRO 4650GE, 6 cores / 12 threads | | RAM | 30 GiB, plus 8 GiB of swap | | GPU | Integrated Radeon RX Vega 6. There is no NVIDIA card. `nvidia-smi` is not installed, and nothing on this box does GPU inference beyond what Ollama does on CPU | | Root filesystem | 295 GB. Disk space is the fleet's most persistent capacity problem. `disk-guard.timer` fires hourly and reclaims a few megabytes per pass. It is running, and it has not solved the problem. A public page cannot read the current figure, so this row gives the command (`ssh nova@ 'df -h /'`) instead of a number | | Reached by | `ssh nova@` | **The bare alias does not resolve.** If you run `ssh` with just `nova` as the host, with no user and no IP, it exits with `Name or service not known`. The Architect box has no `~/.ssh/config` to expand the alias. The working address is always the user at the tailnet IP, which is what the dispatch script itself uses (`NOVA="${DISPATCH_NOVA_IP:-100.x.x.x}"`). This is the most common error in the fleet's internal docs. Thirteen files still write the bare alias, and one of them puts it inside a runbook block meant for copy and paste, so a reader who follows it fails on the first command. ### ETH, the Architect's box {#eth--the-architects-box} ETH is a WSL2 Ubuntu 24.04 guest on the founder's Lenovo, at tailnet `100.x.x.x`, with unix user `kxdev`. `.wslconfig` caps WSL at 12 GB of memory, 4 GB of swap and 6 processors. Inside the guest that shows as about 11 GiB of RAM. This box plans, files issues and dispatches. It hosts no service the fleet depends on. ### utopiamodels, a host that is not a machine {#utopiamodels--a-host-that-is-not-a-machine} This tailnet node is an application that joined the tailnet with `tsnet`. It is an LLM gateway, not a computer. It appears in the node list next to real hardware, serves on port 80, and has no shell. Know this before you try to SSH to it. ## Agents {#agents} **The site generates the list of agents from the registry instead of writing it by hand.** The table below is read from the fleet registry on every request. It used to be typed here, and it was wrong in the way that matters most. It called the DMV-portfolio master `dmv`, a retired alias for the agent the registry records as **`dust`**. That alias has never been a unix account on any box. `id -u dust` returns `1001`, and `id -u dmv` still returns *no such user*. Every `sudo -u dmv` command this site published failed for every reader who copied one, and no page can catch that error in itself. | unix account | box | uid | GitHub identity | backend | registry class | known as | | --- | --- | --- | --- | --- | --- | --- | | `atlas` | novaserver | 1014 | — | Claude | always-on-agent | — | | `cloud` | novaserver | 1002 | `kxagent4` | — | master-agent | — | | `dust` | novaserver | 1001 | `kxagent5` | — | master-agent | — | | `herald` | novaserver | 1015 | — | Claude | always-on-agent | — | | `nova` | novaserver | 1000 | `kxagent3` | — | master-agent | — | | `peaky` | novaserver | 1011 | — | Claude | always-on-agent | — | | `poppi` | novaserver | 1004 | — | Claude | personal-agent | — | | `ronin` | novaserver | 1012 | — | Claude | always-on-agent | — | | `scout` | novaserver | 1005 | — | Claude | always-on-agent | — | | `shred` | novaserver | — | — | Claude | personal-agent | — | | `win` | novaserver | 1003 | `kxagent2` | — | master-agent | — | | `wingman` | novaserver | 1013 | — | Claude | always-on-agent | — | The **unix account** is the name `sudo -u` takes. The **known as** column is filled only where the fleet calls an agent something its account is not — that mismatch is exactly what made every `sudo -u dmv` runbook command on this site fail. **`backend` reads `claude_subscription` off the record, and there is only one value it can take**: the fleet is 100% Claude Code on Claude subscriptions, and `_infra/scripts/dispatch.sh` passes a literal `TASK_BACKEND=claude` with nothing to resolve. A `—` means that record carries no subscription block, not that it runs on something else. *Generated at request time (2026-11-03T05:15:16.823Z) from `utopia-models/knowledge` @ `main` — `wi/registry/organization.yaml` (every `agents:` record — its class, its box, its unix account, its GitHub identity, and whether it carries a subscription block at all); `wi/registry/components.yaml` (the `uid` inside each runtime component’s `where:` line). Nothing here is stored: this page holds no copy of these values.* Each agent has its own credentials, its own `~/.claude` brain directory (each one is its own version-controlled repo), and its own GitHub identity. The agents are peers. No supervisor process exists. The table deliberately leaves out the **binary**. `/usr/bin/claude` is where it resolves on novaserver and nowhere else. On the Architect's WSL box it resolves to a different path. So "every agent runs the same binary at the same path" is a claim about one box presented as a claim about the fleet. To settle it for a given account, run `sudo -u -H command -v claude` on the box where that account lives. ### Existence is not liveness {#existence-is-not-liveness} **The registry checks an agent with `getent passwd `, and that check passes on an agent that cannot execute a single turn.** This is the most important distinction on this page. A master can have a valid unix account, a credential file, a clean systemd state and working `gh` auth, and still be unable to run. Entitlement belongs to the account at the provider, not to anything on the box. When an agent has no entitlement, it looks *busier* than a healthy one. Transient units keep starting, keep dying within seconds, and keep posting their start messages. On 2026-08-16, a measurement found **three of the four masters able to run**. The fourth reported `{"exhausted": true, "resetsAt": null}` in its own state file. The `null` in `resetsAt` is the important part. It means the provider gave no recovery time, so no timer brings that agent back. The dispatch scheduler correctly left it out and said so in its log line every five minutes. The current answer for each agent comes from that agent's own measured entitlement, meaning a recorded refusal and never `getent`. A real `claude -p` turn, run as that agent on its box, settles it for certain. ### There is one backend, and the code that chose between backends was deleted {#there-is-one-backend-and-choosing-between-backends-was-deleted} `dispatch.sh` passes a **literal** backend name into every run, `--setenv=TASK_BACKEND=claude`. It does no lookup, reads no registry and resolves nothing per master. The function that once chose between providers was deleted along with the second provider. This page said the opposite for a while, and the wrong version sounded better. A fleet that resolves each agent's provider at runtime sounds more capable than one that hardcodes a single name. The plain version is stated here because it is what a reader would find in the script. The variable still exists for a different job. It is the fleet's *"am I inside a dispatched run"* signal, and every board-keyed gate uses it to decide when to exempt a run. A flag whose first purpose was deleted can be worth keeping for a second job it took on by accident, as long as someone writes down what that job is. ## The dispatch runtime {#the-dispatch-runtime} The whole agent runtime is two scripts and systemd. There is no scheduler daemon, no queue and no watchdog. This section is the overview. The guards, their exit codes and the race that only a file lock can close are on [The dispatch runtime](/writing/dispatch-runtime). | Script | What it does | | ----------------------------------------------------- | -------------------------------------------------------------- | | `_infra/scripts/dispatch.sh [repo]` | The Architect gives a master its issue **by starting the run** | | `_infra/scripts/task-runner.sh [repo]` | Runs the task itself, on the master's own box | One dispatch does these steps, in order. 1. It launches under the fleet's one backend, which is a literal and not a lookup (see above). 2. `agent` runs locally. Dispatch reaches the novaserver masters with one `ssh nova@` plus `sudo -u `. 3. **Pre-flight refuses a master that is out of quota.** A dispatch to an agent that cannot run wastes the run and charges the issue an attempt for it. The check fails *open*. An unreadable state file, a missing file, or a reset time that has already passed all count as "can run". A false block stops the fleet, and that is worse than a wasted run. 4. **Pre-flight refuses a second run for an issue that is already running.** Each issue gets one run. The guard is in the entry point that both the scheduler and a hand-typed dispatch use, because a guard in only one of them cannot see the other path. 5. **Pre-flight syncs the master's clones.** For every repo in the run's repo list it fetches, stashes anything uncommitted, checks out main and hard-resets to `origin/main`. 6. It launches a **detached transient unit** with `systemd-run --user --unit=task-- --collect`, so the run keeps going if the SSH connection drops or the laptop goes to sleep. 7. `task-runner.sh` updates itself once and checks again that the issue is *still work*. It posts **▶ started** before anything that can fail, then runs `claude -p` under a turn cap. On exit, its trap posts **✔ / ✘ / ⚠ / ⛔** and the pull-request link. Someone may tune the cap in that last step at any time, so a written number would go stale. The page reads it from the source instead: ```bash MAXT="${TASK_MAX_TURNS:-200}" ``` The cap is a **default of 200, not a fixed number** — any caller may raise or lower it by exporting `TASK_MAX_TURNS`, and `_infra/scripts/dispatch.sh` does exactly that when it launches the unit. A run that consumed all its turns reports the cap it actually ran under, which is why the number in a failure message is evidence and the number on this page is not. *Generated at request time (2026-11-03T05:15:16.825Z) from `utopia-models/knowledge` @ `main` — `_infra/scripts/task-runner.sh` (the `MAXT=` assignment passed to `claude -p --max-turns`). Nothing here is stored: this page holds no copy of these values.* ### The last-moment disposal re-check {#the-last-moment-disposal-re-check} An issue can stop being work while its run is starting, and every gate that asks about it runs too early. One real case shows the timing. The pre-flight read the issue at 14:36:01, and it was open and correct. Someone labelled the issue a throwaway at 14:36:03 and closed it at 14:36:05. systemd started the unit at 14:36:12, and the runner posted ▶ at 14:36:15. A master then spent a full session on leftover board items. The fix asks the same two questions from **inside** the run, right before ▶, which is the real point of no return. Only two signals count. Either the issue is CLOSED, or it has the throwaway-fixture label. Any other signal is a judgement that a human may be overriding at the command line on purpose. ### The sweeper {#the-sweeper} The sweeper moves a chain of work forward. It is **not a service and not a cron job**, so every probe that looks for a `sweeper.service` wrongly reports it missing. It runs as the second `ExecStart` line of a systemd drop-in on one master's `reconcile-board.service`, which `reconcile-board.timer` fires every five minutes: ```text /home/nova/.config/systemd/user/reconcile-board.service.d/10-sweep-dispatch.conf ExecStart= (clear) -/usr/bin/bash %h/.claude/ops/reconcile-board-status.sh --apply -/usr/bin/bash %h/dev/utopia-models/knowledge/_infra/scripts/sweep-dispatch.sh --apply -/usr/bin/bash %h/.claude/ops/mute-agent-notifications.sh --apply ``` The sweeper's log line is the best one-line health summary the fleet produces. Here is one real line, read from `~nova/.claude/logs/sweep-dispatch.log` and written at 2026-09-04T08:48:34Z. It is a sample of the format, not a current reading. ```text sweep: 39 open · 0 dispatchable · in-flight 3 (win=2 nova=0 cloud=1 dust=0 ) ``` When the sweeper refuses a master, it adds a trailing segment that names the master and the reason, such as `· rate-limited: (exhausted nova)`. The sample has no such segment because every master was clear on that pass. This line teaches you to read your own logs carefully. A large open count against zero dispatchable looks like a capacity failure, but it is a **routing** failure. Issues with no lane, or on a lane that no sweeper reaches, can never be dispatched, however idle the fleet is. The sweeper printed the cause every five minutes while people investigated a mechanism they assumed was broken. ## Services on the box {#services-on-the-box} The first two groups below run on novaserver and stay up between agent runs. The third group is state the box depends on but does not host. The dispatch runtime has no resident process, but the *box* has plenty. **System units.** `tailscaled`. `cloudflared`, the tunnel that publishes selected local services to the public web. `ttyd`, plus one read-only browser viewport per master on ports 7810 to 7813. `ollama` on 11434, serving small local models. `occ-redirect` on 3939, a small Python responder that sends a 301 from a legacy hostname to the apex. `grafana-server` on 3000. `prometheus` on 9090, with node-exporter on 9100. Two self-hosted GitHub Actions runners, one for a full lane and one for a light lane. **Docker.** **Infisical** runs here (backend, database and redis), and it is the fleet's only secret store. A full **SigNoz** observability stack runs here too, with ClickHouse, a Postgres metastore, keeper and ingester. **Managed Postgres, off the box.** Two hosted databases hold state that nothing here can rebuild, so they belong in this list even though no process for them runs on novaserver. The first is **Neon**, the `utopia-auth` project, Postgres 17 in `aws-us-east-1`. It holds the product's auth and user rows. Every night a `pg_dump -Fc | gzip` job copies it through the direct unpooled endpoint to cold object storage, with 35-day retention. The nightly copy exists because Neon's free point-in-time recovery only reaches back about six hours. The same Neon organisation holds three other projects that are not fleet infrastructure, and it is easy to confuse them with this one. The second is **Supabase**, the `peptide-shop` project in `us-east-1`. It holds the database, auth, and the order and payment records behind `dmvresearch.shop`. Those rows cannot be replaced. Two independent jobs dump them nightly to the same bucket, one with `pg_dump` and one with a row-level export through the REST API. **Scheduling runs on one master.** One master's user manager carries almost every scheduled timer in the fleet. A count on 2026-09-04 with `systemctl --user list-timers --all` under each master account found 37 timers on nova, 4 on win and 3 each on cloud and dust. The important ones publish the command center's feeds, re-render it and check it, sync creative assets, watch token health, refresh the tailnet snapshot, guard the disk hourly, and repair the default route every sixty seconds. **Cron is a second scheduling layer that runs in parallel**, and it is easy to miss. Masters have crontab entries for monitors, self-reports and usage feeds that have no systemd timer at all. An audit that reads only `systemctl --user list-timers` undercounts how often every master really runs jobs. **A dead timer never fails, so `systemctl --failed` reports clean.** The box has had an *enabled* timer whose service unit was `not-found`, and dormant timers that had not fired in a week, while every failure query came back empty. An absence of failures does not mean the thing works. ## Secrets {#secrets} **Infisical is the only secret store.** It is self-hosted in Docker on novaserver. There is no local vault, no `.env` checked in anywhere and no second store. | Operation | Command | | --------------------------- | ------------------------------------------------------------------- | | Read | `bash ~/.claude/scripts/infisical-get.sh [env]` | | List every key in a project | `bash ~/.claude/scripts/infisical-get.sh --keys prod /` | | Write | pipe the value on stdin to `infisical-set.sh [env]` | Projects are split by domain into fleet infrastructure, the product, per-portfolio research and engineering. A guard rejects any value passed as a command-line argument, because a secret in argv lands in shell history and in the process table. A second file, the credential registry, records what each credential *means*. It lists which account the credential belongs to, what scopes it has, what it can do and what it explicitly cannot. A standing rule says to read that registry before you call a credential missing. The rule exists because the fleet's usual failure is not a missing credential. It is a grep for the wrong key name. Two rules have no exceptions. * **Never print a secret, or the fact that one exists, to anything that displays output.** Capture it into a variable and check its length. * **Never write a secret into a tracked file.** This is the fleet's one hard stop. It is the only rule you cannot fix forward, treat as advisory or override. A secret scanner runs on every pull request, and it is the only check whose result the fleet treats as binding. This page publishes the *mechanics* of all of this on purpose, and that is in scope. It never publishes a value.