Over-the-Air Updates¶
A board in someone’s living room has to stay current without anyone touching it. Updates must be signed (an unsigned channel is remote root on every unit), atomic (never a half-installed board), and self-healing (a bad release rolls itself back rather than bricking a customer’s board).
Four moving parts: a CI release bundle, an artifact store, a manifest endpoint, and a board-side updater. The auth is the device token that already existed (Chapter Accounts, Devices and Ratings) — OTA added no new credential.
What ships, and what swaps¶
D-86 — v1 OTA ships a prebuilt, self-contained bundle and swaps the application, not the OS
Context The only way to update a board was to SSH in and compile on the Pi. That is the correct dev-bench tool, and it is not a fleet tool.
Decision CI builds one arm64 bundle per release. The board downloads it; it never carries a toolchain and never compiles. The bundle vendors the Qt runtime — libraries, plugins, the eglfs platform plugin and the QML modules — under one versioned root, loaded by rpath, so the app never links the image’s apt Qt.
Rejected Building on the Pi (slow, and a board that compiles is a board that can fail to compile). A dual-partition A/B rootfs — the appliance gold standard, and real NRE (bootloader integration, twice the storage, an image pipeline) for pre-revenue hand-assembly scale. App-level OTA gets most of the value reusing the service model that already existed.
Consequences Two payoffs from vendoring Qt: Qt itself becomes updatable, and the fleet runs one known Qt rather than whatever apt drift each image carries. Two costs: the bundle grows by tens of megabytes, and CI must build against the golden image’s userspace so the vendored Qt’s glibc and EGL requirements stay satisfiable on-device.
The engine’s release flags use -march=native, which bakes in whatever the CI runner happens to be. The fleet build path pins the Pi 5 core instead (-mcpu=cortex-a76), gated behind a build option so the dev-bench build is untouched.
The honest ceiling: the bundle can ship the app and Qt, but nothing beneath them — not glibc, not the graphics stack, not the kernel, not host config. Those are the image floor (Chapter The Golden Image). Ordinary Qt bumps sit comfortably above it; a Qt demanding a newer glibc or Mesa than the image ships is the reflash trigger.
Signing¶
D-87 — Bundles are signed with minisign, the public key is baked into the image, and nothing unpacks before verification
Context An update channel that a third party can write to is remote root on every board in the fleet.
Decision CI signs each bundle with minisign (Ed25519). The public key is committed in the repository and baked into the golden image, so it is present before the board’s first network call. The updater checks the SHA-256 and then the signature, and unpacks nothing that fails. CI additionally verifies its own freshly-signed artifact against the committed public key, which proves the signing key still matches the key baked into boards.
Rejected Unsigned bundles trusted over TLS alone. TLS authenticates the server, not the artifact — it protects nothing if the store or the manifest endpoint is compromised, which is precisely the scenario worth defending.
Consequences The signature is stored verbatim in the manifest and never trimmed — a minisign blob is newline-significant. The private key is the fleet’s crown jewel: whoever holds it can push root code to every board. A CI secret is the minimum; an offline signer is the real answer before the fleet is more than a handful, and the image should be able to pin more than one public key, because you cannot rotate a key you baked in as a singleton.
The swap, and the safety net¶
/opt/kirin/
releases/
1.4.0/ # unpacked bundle (binary, QML, vendored Qt, net)
1.4.1/ # the new one, staged here
current -> releases/1.4.1 # the ONLY thing that flips
previous -> releases/1.4.0 # kept for rollback
The launcher runs /opt/kirin/current/kirin-gui. An update is: unpack, verify, flip current with a single atomic rename, restart the service. Because the flip is one atomic rename, the board is never half-installed — it is running wholly the old release or wholly the new one.
D-89 — A boot-count guard backs up the heartbeat window for auto-rollback
Context Unattended updating is only acceptable with automatic rollback, and “the process did not exit” is not evidence that a release works.
Decision The GUI writes a heartbeat file once its QML root object is constructed — a positive “this release actually runs” signal, since by then the engine is initialised, the bridges are wired and the UI exists. After a swap the updater watches for that heartbeat inside a bounded window and, failing it, flips current back to the exact prior release and restarts. A boot-count guard — three failed boots on a version — triggers the same rollback.
Rejected Relying on the heartbeat window alone, which a crash loop can slip through if the process survives long enough between restarts.
Consequences A release that cannot construct its UI never sticks. This was exercised live: a deliberately heartbeat-less release auto-rolled back on the bench and the panel recovered.
D-88 — The board never installs on its own — a background check stages, and the owner taps Install
Context The original design had a timer that checked, downloaded and applied.
Decision The timer and the manual check download, verify and stage, then stop, surfacing “update ready”. The atomic swap and restart happen only on an explicit install request from the touch UI.
Rejected Silent auto-update — it “felt predatory for a home appliance”. Notably, the old session-idle gate went away with it: an install is now user-initiated, so there is no surprise mid-game restart to guard against, and the updater no longer reads the lifecycle file for that purpose.
Consequences Staging early makes the later tap instant. If a critical security fix ever needs a background install, it comes back as an explicit opt-in, not as the default.
The updater is a separate service and timer, installed by the provisioning script and baked into the image, deliberately decoupled from the GUI: a crashed GUI must still be replaceable, and an update must not depend on the app being healthy. It is a dependency-light shell agent, and it is deliberately not itself OTA-updated, so it stays simple.
UpdateBridge (Chapter The Application Layer) is correspondingly thin: it reads the updater’s state file and pokes it by writing a verb to a request file that a systemd path unit turns into an updater run. The GUI downloads nothing and swaps nothing — no privilege, no IPC daemon, two small files.
Channels, rollouts and the fleet view¶
The server side is small and additive, and reuses device auth. A device-authenticated GET returns the newest release for the board’s channel, or a 204 if it is current; the board reports its running version on that same call, so fleet version telemetry is free, folded into the presence bump that already existed.
Each board carries a channel (stable by default, beta and dev opt-in), writable only through one admin endpoint — every other path reads it. A release may be published at a partial rollout percentage, and a board is offered it only when its deterministic hash bucket falls inside that percentage; boards outside the rollout fall back to the newest fully-rolled release rather than being stranded on a bare 204. Ramping a release in place preserves its sequence number, so walking 5% $\rightarrow$ 25% $\rightarrow$ 100% never reorders which release is newest — a ramp is not a re-publish.
D-90 — The OTA admin console is built in the corporate admin backend, proxying to the player Worker
Context An operator needs to mint provisioning keys, see every board’s version and channel, and ramp or roll back a release — without a shell.
Decision Build it in web/’s existing admin backend, which already has session-cookie admin auth, a view shell, and — decisively — the exact pattern needed: an admin browser calls the corporate Worker, which relays server-to-server to another service with a shared bearer secret that never leaves the Worker. The OTA console is a near-copy of that, pointed at the player Worker.
Rejected A standalone admin surface in the player Worker. That Worker is purely player-facing and has no admin shell or admin auth at all, so a console there is from scratch. Founder time is the deciding constraint.
Consequences The powerful release and mint secrets stay server-side and never reach a browser. Separation of concerns stays legible: presence comes from the relay, OTA from the player Worker. The console replaced every manual database-query workaround — including minting keys and, later, retiring the stranded device rows an SD re-flash leaves behind.