--- title: "Rebuilding a hand-built server from git" date: 2026-11-06 summary: "How I put a hand-built Linux server under Ansible and OpenTofu, and raised its CIS Level 1 benchmark score from 68.47 to 87.60 without an outage." kind: essay tags: [ansible, opentofu, linux] status: published featured: true order: 1 --- I run a Linux server at home called novaserver, and for most of this year I set it up by hand. When a server is built by hand, nobody can say exactly what's on it, and nobody can build a second one. I'm Tyler, a software engineer at Kratos Defense & Security. novaserver hosts my own projects: a set of Claude Code agents, a few Postgres databases, a self-hosted secret store, monitoring and backups. In October I put the whole machine under code. This post covers what that took, how I hardened it without locking myself out, and the two bugs the hardening turned up. ## Where it started The box had 14 Linux accounts, one per agent plus a service account, each with its own systemd user manager. Accounts, sudo rules, system units and Docker Compose stacks had piled up over months of one-off scripts. When I went to replace the sudo rules I found 18 files in `/etc/sudoers.d` that nothing managed. The goal was one sentence: everything on the box is code, it converges to zero changes, it's tested in CI, and I can see it in monitoring. ## Ansible, tested like code I wrote ten Ansible roles covering the agent accounts, sudo, system units, the Compose stacks, monitoring, hardening and backups. Nine of them have a Molecule scenario. Molecule builds a fresh container, applies the role, then applies it a second time and fails if the second run changes anything. That second run is the real test. One role made 34 changes on the first pass and 0 on the second. ansible-lint at its strictest `production` profile started at 165 findings. It's at 0 now, and CI only runs the Molecule scenarios a change can actually affect, so the checks stay fast. The sudo rules aren't written by hand any more. One template renders them from a single registry file that lists every account, which replaced the 18 unmanaged files. Then the box pulls its own config. A systemd timer runs `ansible-pull` from git every hour, so a change in git reaches the box within the hour, and a hand edit to anything it manages gets put back. ## OpenTofu for everything with an API Anything that lives behind an API went into OpenTofu: Cloudflare DNS and the tunnel, the Tailscale access policy, and the GitHub repositories with their deploy keys. I imported what already existed before writing anything new. That was 32 DNS records, the network policy, 36 repositories and 12 deploy keys, and the import was done when `tofu plan` reported no changes for all three providers. State lives in an S3-compatible bucket with locking and client-side encryption. I checked both: the raw object in the bucket has no readable resources, and a second plan refused to run while the first one held the lock. ## CIS Level 1 without locking myself out I scored the box against the CIS Ubuntu 24.04 Level 1 Server benchmark with OpenSCAP. It started at **68.47**. Hardening a live remote machine has one real risk: you change PAM or SSH, you get it wrong, and you can't log in again. The agents were running the whole time, so I couldn't take the box down either. So I applied the benchmark one section at a time. Before the PAM and SSH section I armed a 15-minute systemd timer that puts the previous config back unless I cancel it. That's a dead-man switch. If a change locks me out, I wait 15 minutes and I'm back in. After every section I checked that no systemd unit had failed and every agent was still running. It finished at **87.60**, with 27 checks still failing. Each of those 27 is a written exception with a reason, and a check confirms the list of failures and the list of exceptions are the same list. Failed units were 0 before and 0 after, and no agent ever stopped. The rollout found two real bugs: - ufw re-applies its own sysctl file, which quietly reset a kernel network setting the hardening role had just set. - A validate-then-reload chain of Ansible handlers could never fire, so the reload it existed for never ran.