Checklist

The goal of Kompismoln is infrastructure that heals and updates itself, and is decentralized so that a server can always be restored from the others.

None of this is new. It's 8 ordinary engineering problems integrated into a whole. Follow along as it happens.

2023

  • May · IaC: A git repo for a workstation's NixOS configuration.
  • Aug: A server in the closet joins: firewall, nginx with https and dynamic DNS.
  • Oct · Keys: Key management declared with agenix.

2024

  • Jan–Feb: The first websites, sverigesval.org, chatddx.com and esse.nu, and a VPS at GleSYS.
  • Mar: A mail server.
  • May–Jun · Network: WireGuard between every server, with the closet server as the gateway for the rest.

2025

  • Feb–Mar: Drive (Nextcloud with Collabora) and the event platform Mobilizon.
  • Aug · Recovery: A server reinstalled with one command. nixos-anywhere installs from bare metal onto encrypted disks, from hardware facts kept with the rest of the configuration.
  • Aug · User data: Data apart from the system. Servers forget everything at reboot but what's declared to be kept.
  • Aug · Keys: agenix → sops, and a first full key rotation.
  • Sep · Keys: An isolated service, "locksmith", and with it a second full key rotation.
  • Oct · Network: More layers of subnets, with a simple mechanism for failover.
  • Oct · IaC: All configuration gathered in org.toml.

2026

  • Jan · User data: A service's data becomes its home, apart from everything else.
  • Jul · IaC: o11n becomes a framework of its own, and the org a file it reads.
  • Sep: One login for everything (Keycloak), and Meet (Jitsi).
  • Oct · Recovery: A USB stick to boot from that contacts Kompismoln by itself.
  • Oct · Keys: Identities apart from keys.
  • Nov · Lifetime: One retention period for every log.
  • Nov · Recovery: Archives on other servers. Snapshots with database dumps are archived daily, encrypted, with restic on another server.
  • Nov · User data: Drive becomes stateless: everything but its data is rebuilt as it starts.
  • Nov: Photos (Immich).
  • Nov · Monitoring: A status page with Gatus, and this site, built with Zola.

8 problems

1. One configuration for everything

Every machine's configuration is generated from the cluster's shared configuration (infrastructure as code).

  • In place with org.toml + o11n

2. User data apart from the runtime environment

What's to be kept is held apart from the runtime environment, which can be rebuilt at any time (erase your darlings).

  • Ephemeral root
  • Coordinated distributed storage, with data on the right machines at once.

3. One master key for all keys

Keys that are declared, derived and rotated without manual work (key management).

  • In place.

4. A network that's both protected and reliable

Separate subnets that survive a route going away and are independent of each other (network segmentation).

  • Layers of redundant hub-and-spoke networks
  • A mesh network for data
  • Drop the rescue mode that's currently on for almost every server and still needed now and then.

5. Recovery that's tested regularly

A server can be reinstalled and get its data back at any time (disaster recovery).

  • Every service's data archived (photos and mail aren't yet).
  • Archives of the archive server, or layers of archives
  • A drill simulated end to end on virtual machines.
  • Regular drills restoring each server from the others (every three months?).

6. Safe continuous updates

New versions are built, applied in order and rolled back automatically if something goes wrong (continuous delivery).

  • git.kompismoln.se: Forgejo with CI/CD, the base for servers that update themselves.
  • Servers fetch and build their own new versions.
  • A server left stranded returns to its last working version on its own.
  • Coordinated updates: a builder pushes the new version to every server, a schedule decides who updates in what order, and the whole cluster rolls back if something goes wrong along the way.

7. Knowing when something breaks

And preferably fixing it (monitoring).

  • A status page that watches continuously
  • Services mail when they fail
  • Alerts when something stops working.
  • Fixing failures automatically.

8. Data with a lifetime

Every piece of data has a set lifetime, and what's erased is gone from every copy within the agreed time (the right to erasure). The requirements come from the policy.

  • One retention period for logs: the journal, access logs, bans and DMARC reports.
  • Drive empties its trash after 30 days and removes older versions after a year.
  • Encrypted disks: a disk is erased by discarding its key.
  • Each service's archive has a key of its own and can be erased in one go.
  • A service's data can be handed back as a file.
  • Archives pruned automatically after the agreed time, and erasures redone after a restore.
  • Retention per service and type of data, declared in org.toml.
  • Erase a person from every service in one step.
  • Traceable access to personal data.
  • A record of processing and of sub-processors, generated from org.toml.

Planned services

  • ai.kompismoln.se: AI for members, on Kompismoln's own machines.
  • maps.kompismoln.se: maps.
  • calendar.kompismoln.se: calendar.
  • internet-archive.kompismoln.se: copies of parts of the internet, such as Wikipedia, for the apocalypse.