I run a Linux server at home called novaserver, and for most of this year I set it up by hand. When a server is built by hand, nobody can say exactly what's on it, and nobody can build a second one.
I'm Tyler, a software engineer at Kratos Defense & Security. novaserver hosts my own projects: a set of Claude Code agents, a few Postgres databases, a self-hosted secret store, monitoring and backups. In October I put the whole machine under code. This post covers what that took, how I hardened it without locking myself out, and the two bugs the hardening turned up.
Where it started
The box had 14 Linux accounts, one per agent plus a service account, each with its own systemd user manager. Accounts, sudo rules, system units and Docker Compose stacks had piled up over months of one-off scripts. When I went to replace the sudo rules I found 18 files in /etc/sudoers.d that nothing managed.
The goal was one sentence: everything on the box is code, it converges to zero changes, it's tested in CI, and I can see it in monitoring.
Ansible, tested like code
I wrote ten Ansible roles covering the agent accounts, sudo, system units, the Compose stacks, monitoring, hardening and backups. Nine of them have a Molecule scenario. Molecule builds a fresh container, applies the role, then applies it a second time and fails if the second run changes anything. That second run is the real test. One role made 34 changes on the first pass and 0 on the second.
ansible-lint at its strictest production profile started at 165 findings. It's at 0 now, and CI only runs the Molecule scenarios a change can actually affect, so the checks stay fast.
The sudo rules aren't written by hand any more. One template renders them from a single registry file that lists every account, which replaced the 18 unmanaged files.
Then the box pulls its own config. A systemd timer runs ansible-pull from git every hour, so a change in git reaches the box within the hour, and a hand edit to anything it manages gets put back.
OpenTofu for everything with an API
Anything that lives behind an API went into OpenTofu: Cloudflare DNS and the tunnel, the Tailscale access policy, and the GitHub repositories with their deploy keys. I imported what already existed before writing anything new. That was 32 DNS records, the network policy, 36 repositories and 12 deploy keys, and the import was done when tofu plan reported no changes for all three providers.
State lives in an S3-compatible bucket with locking and client-side encryption. I checked both: the raw object in the bucket has no readable resources, and a second plan refused to run while the first one held the lock.
CIS Level 1 without locking myself out
I scored the box against the CIS Ubuntu 24.04 Level 1 Server benchmark with OpenSCAP. It started at 68.47.
Hardening a live remote machine has one real risk: you change PAM or SSH, you get it wrong, and you can't log in again. The agents were running the whole time, so I couldn't take the box down either.
So I applied the benchmark one section at a time. Before the PAM and SSH section I armed a 15-minute systemd timer that puts the previous config back unless I cancel it. That's a dead-man switch. If a change locks me out, I wait 15 minutes and I'm back in. After every section I checked that no systemd unit had failed and every agent was still running.
It finished at 87.60, with 27 checks still failing. Each of those 27 is a written exception with a reason, and a check confirms the list of failures and the list of exceptions are the same list. Failed units were 0 before and 0 after, and no agent ever stopped.
The rollout found two real bugs:
- ufw re-applies its own sysctl file, which quietly reset a kernel network setting the hardening role had just set.
- A validate-then-reload chain of Ansible handlers could never fire, so the reload it existed for never ran.