# The always-on agent

Tyler Youk · updated 2026-10-31 · https://tyleryouk.com/writing/always-on-agent

Everything else in this fleet is event-driven. Work arrives on a board, a run announces itself on an
issue, and a merge changes a status. The fleet forbids a resident process for anything that the board, a
dispatched run or a git event already announces. Heartbeat polls and scheduled health checks fall in that
group. When they fail they go silent, and silence looks exactly like health.

There is one exception, and this page is about why it was allowed.

## What a permanent loop has to prove {#what-a-permanent-loop-has-to-prove}

The amendment permits a resident process only to watch an external system that emits no signal anyone
can subscribe to, and only where reacting faster than the polling interval is the actual goal. It adds one
requirement that does most of the work.

The process must name the external event it exists to catch, in its own documentation.

That requirement is the part other teams can reuse. Suppose a daemon watches for an event nobody can subscribe
to. If it cannot name that event, it is polling something it could have been told about. Writing the name down
makes that obvious to the author, before anyone else has to review it.

This process names two events:

* **A role appearing on an applicant-tracking board.** No webhook, feed or subscription exists for *"a
  role opened at any company in this region"*. The only advantage is being early, and an application sent
  after two minutes is a different application from one sent after twelve hours.
* **A chat-membership update at bootstrap.** This is the moment a human adds the bot to a group. This watch
  is finished. The poll stopped for good once it captured the identifier, and the published feed reports it
  as captured, not as running. A watch should stop once it has caught its event. A watch that keeps polling
  for something it already has is built wrong.

## The loop {#the-loop}

```text
service, Restart=always            one cycle
  ├─ detect     watch the external boards
  ├─ enrich     fill in what was seen
  ├─ mailbox    inbound replies
  ├─ notify     outbound, successes only
  └─ apply      act on what detect wrote   ← LAST, and deliberately

each duty runs every executable in its directory, sequentially,
logging the name and the exit code of each

handlers WRAP the canonical tools in the main repository — never a fork

every cycle merges a liveness block into a state file
  → collected into a committed feed
    → rendered on the command center, where a human already looks
```

## Three properties that are load-bearing and easy to undo {#three-properties-that-are-load-bearing-and-easy-to-undo}

### A duty is registered in the loop, not discovered from its directory {#a-duty-is-registered-in-the-loop-not-discovered-from-its-directory}

The loop body lists, as literal calls, *which* directories run. If someone drops a handler into a
directory that is not on that list, the handler looks installed but is dead code. It is present and
executable, and nothing ever calls it.

The tempting alternative is to discover handlers by scanning directories. That buys a small convenience
and costs a failure that nobody sees. With an explicit list, adding a handler takes one edit that a
reviewer can see.

The last duty runs last for a measured reason. It is the only handler that can take minutes instead of
seconds. Across 45 records its median is 19 seconds, its 90th percentile is 98 and its maximum is 246. The
duties run one after another, so if it ran before the watch, it would use up the latency budget that the
process exists to protect.

In a loop that runs duties in sequence, order matters, so write the reason for the order next to the order.

### The restart limit is what makes death visible {#the-restart-limit-is-what-makes-death-visible}

`Restart=always` on its own restarts a crash loop forever. The unit never enters a failed state, the
failure handler never fires, and the always-on agent dies while its service manager still reports it as
active.

A start-limit interval and burst turn an endless crash loop into a unit that eventually gives up, enters
`failed`, and triggers the notification path. Without them, the design loses its most important property,
that failure announces itself. It loses it for the one component that has no other liveness signal, and
nothing tells you.

### A schedule is a timer. Only the watch is resident {#a-schedule-is-a-timer-only-the-watch-is-resident}

The daily rollup that summarizes what the loop did is a timer, not a sixth duty. There are three reasons,
and you should test any future addition against each one:

1. A rollup of rows already written gains nothing from low latency, so it fails the test that allowed the
   resident process in the first place.
2. The duties run in sequence, so a rollup inside the loop would use up the watch's time budget.
3. A report on what the system did today must keep working when the thing it reports on stops.

Here, anything that runs on a schedule is a timer, and only the watch is resident. That is the test for any
proposed sixth duty, and it is much easier to apply than "is this important enough."

## Its isolation is enforced by code that never reads the registry {#its-isolation-is-enforced-by-code-that-never-reads-the-registry}

The scheduler's list of dispatchable agents is a hardcoded string in two scripts. This agent is in
neither, so no board issue can route to it. A misconfiguration, a typo in a lane name, or someone assigning
it in good faith cannot change that.

Adding it to either list would make an always-on agent dispatchable, and that is the one property agents
of this kind must not have. So the isolation is not a policy that anyone has to remember. It is a name
missing from two string literals, and that guarantee keeps holding even after everyone forgets about it.

The second half is simple. This agent never runs in the foreground. All human interaction with it goes
through the planning agent. If the resident process had its own chat interface, people would have a second
place to give it instructions, and a second place to give a system instructions is a second tracker by
another name.

## Where this one is currently weakest {#where-this-one-is-currently-weakest}

* The unit files are not in any repository. A script installs them from a different repository, and the
  main repository holds neither the files nor the script, so nothing compares the running schedule against
  a committed artifact. Read the live unit files on the box. Do not trust a written copy, including this
  page.
* The interpreter is a virtual environment on the box, and nothing in git describes it. One duty needs a
  real browser under a virtual display, and on this distribution the system interpreter has no package
  directory of its own. Three commands documented in the agent's own brain repository create the
  environment. When it is missing, the handler's pre-flight check names it, instead of failing with a bare
  non-zero exit. Check that it exists. Do not assume it.
* A probe run as the wrong user reports a missing unit and a unit it cannot see in the same way. These are
  per-user service units, so a probe from another account cannot see them, and the failure text does not
  tell "not there" apart from "not yours to see." Run the probe as that account with the runtime directory
  set. Otherwise read the committed feed, which carries its own timestamp and is the only reading available
  from an account that cannot reach the box at all.

That last point applies well beyond this fleet. A health check that cannot tell "absent" from "not
permitted" will eventually report a working system as dead.