# Decisions

Tyler Youk · updated 2026-10-31 · https://tyleryouk.com/writing/decisions

The fleet's most expensive recurring failure is deciding again something it already decided and paid
for. Each entry below says what the thing was, why it died, and what replaced it. If you want to propose
one of these again, bring new information. A fresh opinion is not enough.

Disagreement does not reopen a decision. A superseding record that names what changed does. You write it
beside the original and update the original's `Superseded-by` header in the same commit. A decision
record that someone silently invalidated is worse than no record.

## Killed: ralph, the 24×7 autonomous loop {#killed-ralph--the-247-autonomous-loop}

**Deleted 2026-07-24.**

`ralph.service` ran with `Restart=always` and drove a loop. It picked the next open board issue for this
master's lane, ran headless `claude -p`, and repeated. Over time it collected watchdogs, heartbeats,
priority-sort tests, self-reload fixes and a pick-order bug. All of that supervision code existed only to
keep one resident process honest.

On-demand dispatch replaced it. `dispatch.sh` plus `task-runner.sh` do the work, and no agent process
stays resident. The close-out measured the replacement instead of asserting it. It recorded **7
dispatched runs, 7 verified outcomes, 0 runtime failures**, against a baseline of roughly 20%.

Clearing what it left behind took a second deletion. For weeks after we deleted the loop, the registry
still had a per-agent field named after it, still described as the live backend selector. Code really
did read the field, so nobody could call it dead. The description outlived the system, and then the
field outlived the description. Both are gone now. Nothing in the dispatch scripts selects a backend, and
the launcher passes one literal provider name to every run.

A name that outlives its system keeps costing you. Everyone who reads the surviving name afterwards
concludes the deletion is incomplete.

## Killed: the generated component catalog and its fail-closed gates {#killed-the-generated-component-catalog-and-its-fail-closed-gates}

**Deleted 2026-07-23/24.**

This was an 804-line machine-generated registry. Every component in it carried a shell predicate, and
three separate workflows plus a post-commit regeneration hook ran those predicates **fail-closed on every
pull request**. Its own header claimed *"a fix isn't done until catalog = reality."*

We killed it because a gate that blocks on a generated map **blocked documentation changes that were
fine, and reverted its own cleanup automatically**. One commit removed eighteen dead gate workflows.

The replacement reverses the design on purpose. One component registry exists, maintained by hand. Every
entry has a `verify:` command that **you run**, and the file tells you to run it and never trust its
prose. Nothing enforces it, by design.

This wiki publishes the unflattering half too. An audit found that humans read `verify:` and nothing
executes it. It also records no box, so a quarter of its commands fail from the wrong machine and pass
from the right one. The new registry is better than the catalog it replaced, but it does not yet do what
its own header claims.

## Killed: the NATS message bus {#killed-the-nats-message-bus}

`nats.service` and its reload units are on the **never resurrect** list. Issue comments and @mentions
replaced it for cross-machine agent messages. That delivery is durable and at-least-once, and it shows up
where a human already reads.

**Three leftover units remain on the box, and one of them is active.** The file that documented them
was itself deleted, so we name them here. This page used to say there were *two*, and that they were
*inactive and disabled*. Then a reader doing the pattern walk ran the command and reported otherwise
(knowledge#6665). We measured again on the box on 2026-08-23:

```console
$ systemctl list-unit-files | grep -i nats
nats-reload.path       enabled   enabled
nats-reload.service    static    -
nats.service           disabled  enabled

$ systemctl list-units --all | grep -i nats
nats-reload.path      loaded active   waiting  Watch NATS config; reload server on change
nats-reload.service   loaded inactive dead     Reload NATS on config change
nats.service          loaded inactive dead     NATS JetStream fleet hub
```

`nats-reload.path` is `enabled` and `active (waiting)`. It is a live path unit watching the config of a
service that will never start again. Deleting something in a repository is half of the deletion, and the
other half is the part that keeps running.

## Killed: the command center's generated infrastructure surface {#killed-the-command-centers-generated-infrastructure-surface}

This view was built on the deleted catalog, so once the catalog died it showed a permanent empty state.
A single pull request removed it, deleting *"11 callerless routes, 7 orphaned components, 5 dead libs."*

It is listed here because **open plans still name its deletion as a to-do.** Anyone dispatched on that
line finds nothing to delete, and that looks like a lost change rather than one that already shipped. A
stale local clone may disagree. Trust `git ls-tree origin/main`, not a local `ls`.

## Killed: a scheduled local re-run of another repository's CI {#killed-a-scheduled-local-re-run-of-another-repositorys-ci}

**Deleted 2026-08-15.**

A timer on one master fired every three hours. It checked out the product repo, ran typecheck, build and
test, and filed a P0 on any real failure. Someone built it to cover a real blind spot, and people
repaired it many times over months.

It died because the product repo's own CI runs those same three phases on every push to main. That CI
simply **told nobody**. Once the CI announced its own results, a second box re-running the same work was
wasted effort.

### The half-deletion, which is the real lesson {#the-half-deletion-which-is-the-real-lesson}

**Deleting the unit from the agent's brain repository did not stop it running.** `~/.config/systemd/user/`
holds *copies* of unit files, not symlinks into the repo. The timer stayed `active waiting`, and its
`ExecStart` pointed at a script that no longer existed. The next firing came nearly three hours after
the "delete". It exited `203/EXEC` and filed a P0 about itself.

Deleting from the brain repo is **half** of a deletion. The other half happens on the box and takes five
commands. Operations lists them.

## Standing decisions {#standing-decisions}

These decisions are current, not history, and they constrain new work.

### Publish the mechanics, not the identifying values {#publish-the-mechanics-not-the-identifying-values}

**Superseded 2026-09-27.** This entry used to say *"publish everything, unsanitized"*. That meant real
hostnames, real tailnet IPs and real usernames. We decided it after measuring what that exposed, and the
entry said outright that a contributor who sanitized the site was causing a regression. The founder
narrowed the rule because the audience changed. People now copy these pages into their own agents as
prompts. For that reader, a literal tailnet address or an admin UI's hostname teaches nothing that a
placeholder does not.

The current rule is to **publish how everything works and withhold the literal values that locate
private infrastructure.** Tailnet addresses become `100.x.x.x` or `<tailnet-ip>`. Internal hostnames and
bucket URLs become named placeholders. Every row, class and lesson around them stays the same. We apply
the redaction one page at a time. Surfaces and
[Architecture](/writing/architecture) have it, and so does the tailnet table on the landing page.

This change also follows the reopening rule at the top of this page. We did not silently delete the old
entry. We marked it superseded here and said what changed.

**The one hard stop has not changed and has no exceptions. Never put a secret in a tracked file.**
Publishing how the vault works is in scope. Publishing a credential never is.

### One secret store {#one-secret-store}

The fleet uses self-hosted Infisical and nothing else. There is no local vault, no second store and no
`.env` in a repository. A separate registry records what each credential *means*, meaning its account,
its scopes, and what it can and cannot do. That way, anyone who says "I don't have access" has to check
the claim against a list first.

### Subtract by default {#subtract-by-default}

Before you add a script, a rule, a service, a gate or a file, ask what you can **delete or merge
instead**. When something replaces legacy code, delete the legacy code at that moment. Git history is
the archive, and `archive/` folders are banned. A convention gets a linter that only warns. Only an
invariant that has stopped changing gets a gate that blocks.

### Options before decisions {#options-before-decisions}

Before a non-trivial architecture or tooling choice, research the alternatives. That means at least two
real options, plus doing nothing, plus using something that already exists. Then write the decision
down. Proven native and open-source tools beat custom code. Custom code is the exception, and it has to
justify itself.

This site is an example. The wiki lives in its own repository rather than inside the product monorepo.
Before anyone wrote the first line of the app, we committed the record that names the four options
considered, the reason for the choice, and the **costs the choice imposes**.

### A limit is a measurement, never a precaution {#a-limit-is-a-measurement-never-a-precaution}

Before you set any ceiling, such as concurrency, retries, batch size or timeout, measure the real
headroom and **write the measurement next to the number**. A cap chosen out of caution quietly becomes
the system's capacity. Unlike a bug, it never shows itself. It reports success while doing a fraction of
the work.

The fleet learned this the annoying way. Its dispatch scheduler ran for days at roughly **8% of proven
capacity**. Three of four masters sat idle on a 12-core box with a load average below 1. The cause was
caps of 1, 4 and 2, set against a failure mode that two other mechanisms already covered. The defence
was real, but the caps were the wrong place to put it.

### Fix forward, never pre-gate {#fix-forward-never-pre-gate}

Build, merge and deploy. If something breaks, revert it or fix it and redeploy. We measured the
alternative, a human approving each step, and it is slower at every scale this fleet runs at. This rule
does *not* allow skipping verification, force-pushing shared branches, or claiming a result nobody
observed.

## Never resurrect {#never-resurrect}

* **ralph** and every watchdog, heartbeat and supervisor built to watch over it.
* **The NATS bus** and its three remaining leftover units.
* **The generated component catalog** and any fail-closed gate built on a generated map.
* **Content-keyword gates** of any kind.
* **Resident processes, heartbeat polls and scheduled health checks** as a way to know whether the
  fleet is alive. The board is the queue. A run's own close-out comment is the liveness signal. The one
  page that reads current state reads it at request time and stores nothing.

  **There is one sanctioned exception, and we name it here so you do not have to find it with `ps`.**
  Each master's own crontab has a `*/5 * * * *` entry that POSTs that master's record to the command
  centre. The command centre stores it with a five-minute TTL. When the TTL expires, the box turns grey
  on a dashboard a human is watching. That is a self-reported heartbeat, and we adopted it on purpose
  (2026-07-22) because it answers a different question. It tells you *which box is online right now*,
  not *whether this work is alive*. Nothing in the work loop reads it. No dispatch, gate or close-out
  depends on it, and that keeps the ban meaningful. Separately, a master's own standing interactive
  session stays resident until someone restarts it (measured at 13.3 days on one master, 2026-08-23).
  No *dispatched run* leaves a process behind. The ban is on resident machinery as the way to answer "is
  the fleet alive". It has never banned every scheduled job. A reader who runs the walk from inside this
  fleet will find both of these, so we wrote them down rather than let them look like drift
  (knowledge#6665).

If you think one of these should come back, write a superseding decision record that names what
changed. Do not rebuild it.