# The Install Tree — a backend-neutral service-install abstraction for Skíðblaðnir

*One declarative `ServiceSpec` → every modern (2015+) Linux/embedded init & service*
*system, with a multi-OS KVM install **test matrix** and a machine-injected*
*introspection **probe** that reports "installed + started + healthy" as DATA.*

> **Status (2026-06-30):** design + landed spike. The spike (branch
> `agent/install-tree-spike`, worktree `/home/rickard/git/install-tree-spike-wt`)
> ships `src/service.rs`: the neutral `ServiceSpec` + the `ServiceRenderer` trait +
> **two real renderers** (systemd system/`--user` and OpenRC). `cargo test` green
> (82 passed), clippy clean. Everything below the spike (the other 5 init renderers,
> the test matrix, the probe) is design.

---

## 0. Why a tree

`src/systemd.rs` is excellent — but it is **one backend**. Its `SystemdServiceSpec`
renders straight to a systemd `.service` unit and nothing else. Today every
nordisk product (nornir, holger) couples its install story to systemd. That is
fine on Ubuntu/Fedora/RHEL, but Skíðblaðnir's whole pitch is *"airgap a whole
system across a gap with no network, no shell, no manual steps"* — and the systems
you airgap onto in 2026 are not all systemd:

- **Alpine / postmarketOS / embedded** → OpenRC (and increasingly **dinit**).
- **Void Linux / many appliance images** → runit.
- **Gentoo / hardened / s6 shops** → s6 + s6-rc.
- **Artix / Devuan / Chimera** → OpenRC | runit | s6 | **dinit**.
- **Ancient/locked RHEL6-era or busybox** → SysV fallback.
- **Container hosts** → OCI restart policies / Podman **quadlet** / `systemd-nspawn`.
- **Windows targets** → the SCM (already half-covered by `msi::WindowsServiceSpec`).

Hard-coupling to systemd re-creates exactly the per-init bespoke install scripts
Skíðblaðnir set out to delete (the `x14-datalabb` Ansible problem). The fix is a
**tree**: one declarative spec at the root, one renderer per backend at the leaves.

---

## PART A — the modern service/init landscape (research)

A 2015+ install tree must cover the following. Each row is what a renderer has to
emit and how the lifecycle verbs map. (The right-hand "abstraction" column is what
§B's `ServiceSpec` distils.)

### A.1 systemd — system scope

- **Artifact:** INI unit `/etc/systemd/system/<name>.service` with `[Unit]`
  (`Description=`, `After=`, `Wants=`/`Requires=`), `[Service]` (`Type=`,
  `ExecStart=`, `User=`/`Group=`, `Environment=`/`EnvironmentFile=`, `Restart=`,
  `RestartSec=`, hardening: `NoNewPrivileges=`, `ProtectSystem=`…), `[Install]`
  (`WantedBy=multi-user.target`).
- **Verbs:** `systemctl daemon-reload`, `enable`, `start`, `is-active`/`status`,
  `stop`, `disable`. Skíðblaðnir issues these over **D-Bus** (`org.freedesktop.systemd1`,
  the existing `systemd::Systemd` zbus client) — no `systemctl` subprocess.
- **Scope:** system (root, dedicated user).
- **Logging:** journald (`journalctl -u <name>`).
- **Ordering:** `After=`/`Before=` (ordering) vs `Wants=`/`Requires=` (pull-in).
- **Socket activation:** a sibling `<name>.socket` unit (`ListenStream=`) + `Type=notify`.
- *Already implemented* by `src/systemd.rs`.

### A.2 systemd — `--user` scope

- **Artifact:** same INI, in `~/.config/systemd/user/<name>.service`; **omit**
  `User=`/`Group=` (the unit already runs as you — systemd rejects them);
  `[Install] WantedBy=default.target`.
- **Verbs:** `systemctl --user …`, over the per-user **session** bus.
- **Scope:** the invoking login user; no root, no `/etc`. State under `$HOME`.
- *Already implemented* by `src/systemd.rs` (`Scope::User`).

### A.3 OpenRC

- **Artifact:** an `#!/sbin/openrc-run` shell-ish service script in
  `/etc/init.d/<name>`; settings as variables (`command=`, `command_args=`,
  `command_user=`, `pidfile=`, `directory=`, `command_background=true`). Env is
  auto-sourced from `/etc/conf.d/<name>`. Ordering/needs live in a `depend()`
  function (`need net`, `after <svc>`, `use <svc>`).
- **Restart supervision:** modern OpenRC uses **`supervise-daemon`**
  (`supervisor="supervise-daemon"`, `respawn_delay=`, `respawn_max=`) — the
  `Restart=`/`RestartSec=` twin.
- **Verbs:** `rc-update add <name> default` (enable at boot), `rc-service <name>
  start|status|stop`.
- **Scope:** system; user services exist (`/etc/init.d/.../user`) but are rare.
- **Logging:** stdout/err to syslog or `output_log=`/`error_log=`.
- **Socket activation:** none native (the service binds its own port).

### A.4 runit

- **Artifact:** a service **directory** `/etc/sv/<name>/` with an executable
  `run` script (`exec chpst -u <user> <cmd> 2>&1`), optional `finish`, and a
  `log/run` for the logging chain (`svlogd`). Enable = symlink into the supervised
  dir `/var/service` (or `/run/runit/service`).
- **Restart:** automatic — runit *always* respawns `run` (to NOT restart you exit
  the script or use a `down` file / `check`). `RestartPolicy::No` ⇒ a `run` that
  execs once and a `down` marker.
- **Verbs:** `ln -s /etc/sv/<name> /var/service` (enable), `sv up|status|down <name>`.
- **Scope:** system (per-user via `runsvdir` in a login session is possible).
- **Logging:** the `log/run` → `svlogd` directory chain.
- **Ordering:** no declarative graph — encode via `sv check`/`sv start <dep>` at the
  top of `run`, or a oneshot ordering service. (The renderer emits a `run` that
  blocks on deps with `sv check`.)
- **Socket activation:** none native.

### A.5 s6 / s6-rc

- **Artifact (source form):** a **definition directory** per service holding a
  `type` file (`oneshot` | `longrun` | `bundle`), a `run` executable for longruns
  (`#!/bin/execlineb` or shell), optional `finish`, `notification-fd` (readiness),
  and a `dependencies` file (one dep service per line). A `bundle` groups services
  via a `contents.d/` dir. The source tree is **compiled** with `s6-rc-compile` into
  a read-only `compiled` DB.
- **Verbs:** `s6-rc-compile <compiled> <source>` (build), `s6-rc-update` (swap live
  DB), `s6-rc -u change <bundle>` (start/up), `s6-rc -d change <bundle>` (down),
  `s6-svstat` (status).
- **Restart:** s6-supervise always restarts a longrun; tune with `down-signal`,
  `max-death-tally`, a `finish` that exits non-zero to stop.
- **Scope:** system (per-user supervision trees are idiomatic for s6).
- **Logging:** a dedicated `<name>/log` longrun piping to `s6-log`.
- **Ordering:** the `dependencies` file (hard) — s6-rc never starts A before B is up.
- **Socket activation:** via `s6-ipcserver`/`s6-tcpserver` superservers in `run`.
- *Note:* s6-rc needs a **compile step**, so its renderer emits a source tree +
  records `s6-rc-compile` as a build verb (the only backend with a compile phase).

### A.6 dinit

- **Artifact:** a textual service description in `/etc/dinit.d/<name>` (system) or
  `/etc/dinit.d/user` (user). Keys: `type = process | bgprocess | scripted |
  internal | triggered`, `command = …`, `stop-command =`, `run-as =`, `restart =
  yes|no`, `pid-file =`, `logfile =`, `working-dir =`, `env-file =`, and three
  dependency kinds: `depends-on:` (hard — stop-propagates), `depends-ms:`
  (milestone — needed to start, but not stop-propagated), `waits-for:` (soft —
  may fail).
- **Verbs:** `dinitctl enable <name>` (and `start|status|stop|disable`).
- **Restart:** `restart = yes` + `restart-delay`/`restart-limit-count`.
- **Scope:** system + user (`/etc/dinit.d/user`, `dinitctl --user`).
- **Logging:** `logfile =` or console.
- **Ordering:** `depends-on`/`depends-ms`/`waits-for` — the **richest** 3-tier dep
  model of the lot (maps cleanly to `needs` vs `after`).
- **Socket activation:** not native; `triggered` type covers event-start.

### A.7 SysV init (the universal fallback)

- **Artifact:** an LSB `/etc/init.d/<name>` shell script with an
  `### BEGIN INIT INFO … ### END INIT INFO` header (`Provides:`,
  `Required-Start:`, `Default-Start: 2 3 4 5`) and `start|stop|restart|status`
  cases driven by `start-stop-daemon`.
- **Verbs:** `update-rc.d <name> defaults` / `chkconfig <name> on` (enable),
  `service <name> start|status|stop`.
- **Restart:** no supervision — needs a `respawn` line in `/etc/inittab` or a
  watchdog. `RestartPolicy` is best-effort here (documented limitation).
- **Scope:** system only. **Logging:** whatever the script redirects.
- **Ordering:** the LSB header's `Required-Start:` + runlevel symlink order.
- This is the lowest-common-denominator target; the renderer is the "it'll always
  boot *something*" safety net.

### A.8 OCI / Docker

- **Artifact:** not a file on disk but a **container run spec** — image ref, env,
  published ports, mounts, and a **restart policy** (`--restart=on-failure:5` /
  `always` / `unless-stopped`) the runtime's own supervisor honors. The unit fields
  map to `docker run`/`podman run` flags or a compose service.
- **Verbs:** `podman create|start`, `podman ps`/`inspect` (status), `stop`.
- **Scope:** system or rootless (rootless podman ≈ the `--user` analogue).
- **Logging:** `podman logs` (json-file / journald driver).
- **Ordering:** compose `depends_on` / pod start order. **Socket activation:**
  systemd socket → container, or the app binds inside.

### A.9 Kubernetes-adjacent — Podman quadlet / systemd-nspawn

- **Artifact:** a **quadlet** `<name>.container` file in
  `/etc/containers/systemd/` (or `~/.config/containers/systemd/`) that
  `podman-systemd` (the `podman-system-generator`) renders into a transient
  systemd `.service` at daemon-reload. Keys: `[Container]` (`Image=`, `Exec=`,
  `PublishPort=`, `Environment=`, `User=`), `[Service]` (`Restart=`), `[Install]`
  (`WantedBy=`). This is the clean bridge "declarative container ⇒ systemd-managed".
- **Verbs:** `systemctl daemon-reload` then `systemctl start <name>` (the generated
  unit). **systemd-nspawn:** `machinectl start <name>` over a container image in
  `/var/lib/machines`.
- This is the highest-leverage container target because it **reuses the systemd
  renderer's verbs** — a quadlet *is* a systemd unit once generated.

### A.10 Windows service (SCM) — for completeness

- **Artifact:** an SCM registration (the existing `msi::WindowsServiceSpec`:
  account `LocalSystem`, `Start = Auto|Manual|Disabled`, delayed-auto, env block in
  `HKLM\SYSTEM\CurrentControlSet\Services\<name>`). Built into the MSI by WiX.
- **Verbs:** `sc create|start|query|stop` / `New-Service` (PowerShell);
  `msiexec /i … /qn` for the packaged install.
- **Scope:** machine (LocalSystem) or a named service account.
- **Logging:** Windows Event Log. **Ordering:** `DependOnService`.

### A.11 The cross-cutting abstraction these all share

Strip the syntax and **every** backend is the same six facts plus verbs:

| Neutral concept | systemd | OpenRC | runit | s6-rc | dinit | SysV | OCI/quadlet | Windows |
|---|---|---|---|---|---|---|---|---|
| **what to run** | `ExecStart=` | `command=`+`command_args=` | `run` exec | `run` | `command=` | `start)` case | `Exec=`/cmd | service binary |
| **as whom** | `User=`/`Group=` | `command_user=` | `chpst -u` | `s6-setuidgid` | `run-as=` | `--chuid` | `User=`/rootless | account |
| **scope** | system / `--user` | system | system | system/user | system/user | system | system/rootless | machine/account |
| **env** | `Environment=`/`EnvironmentFile=` | `/etc/conf.d/<n>` | `chpst -e` dir | `s6-envdir` | `env-file=` | export in script | `Environment=` | env block |
| **ordering** | `After=` | `depend(){after}` | `sv check` | `dependencies` | `waits-for:` | `Required-Start:` | `depends_on` | `DependOnService` |
| **hard need** | `Requires=`+`After=` | `depend(){need}` | block in `run` | `dependencies` | `depends-on:` | `Required-Start:` | pod/compose | `DependOnService` |
| **restart** | `Restart=`/`RestartSec=` | `supervise-daemon`+`respawn_delay` | always (default) | always (default) | `restart=`+delay | inittab respawn | `--restart=` | recovery actions |
| **status verb** | `is-active` | `rc-service status` | `sv status` | `s6-svstat` | `dinitctl status` | `service status` | `podman inspect` | `sc query` |

That table **is** the `ServiceSpec`. One declarative spec carries the left column;
each renderer knows its own row.

---

## PART B — the tree design

### B.1 Shape

```
                         ServiceSpec                (one declarative spec; serde)
                              │
                     ServiceRenderer (trait)        (pure: spec -> RenderedService)
        ┌──────────┬─────────┼─────────┬──────────┬─────────┬──── … ────┐
   systemd-system  systemd-user   OpenRC      runit     s6-rc     dinit   SysV/OCI/quadlet/Win
   [IMPLEMENTED]   [IMPLEMENTED]  [IMPL.]    [declared][declared][declared]   [declared]
        │              │            │
   bridges existing systemd.rs   native render
                              │
                       RenderedService              (files: Vec<RenderedFile> +
                              │                       enable/start/status/stop argv DATA)
                       apply()  (privileged driver — write files, run verbs; follow-up)
```

The tree is a strict **generalization** of `src/systemd.rs`, not a replacement:
the systemd renderer *bridges* to the existing tested `systemd::systemd_unit_scoped`,
so the install limb that already ships is reused, never forked. `ServiceSpec`
reuses `systemd::{Scope, RestartPolicy}` as shared vocabulary.

### B.2 The types (landed in the spike, `src/service.rs`)

```rust
pub enum Backend {                 // the full 2015+ tree (10 leaves)
    SystemdSystem, SystemdUser, OpenRc,           // implemented
    Runit, S6Rc, Dinit, SysV, Oci, Quadlet, WindowsService,  // declared
}
impl Backend { fn all() -> &'static [Backend]; fn slug(self) -> &'static str; }

pub struct ServiceSpec {           // ONE declarative spec
    name, description, exec, args,
    user: Option<String>,          // None ⇒ run as caller (required for Scope::User)
    scope: Scope,                  // System | User  (reused from systemd)
    working_dir: Option<String>,
    env: Vec<(String,String)>,
    after: Vec<String>,            // ordering edges
    needs: Vec<String>,            // hard requires
    restart: RestartPolicy,        // No | OnFailure | Always (reused)
    restart_sec: u32,
    sockets: Vec<Socket>,          // socket activation / declared ports
}

pub struct RenderedFile { path: String, contents: String, mode: u32 }
pub struct RenderedService {       // the render output: files + verbs AS DATA
    backend: Backend,
    files: Vec<RenderedFile>,
    enable: Vec<String>, start: Vec<String>, status: Vec<String>, stop: Vec<String>,
}

pub trait ServiceRenderer { fn backend(&self)->Backend; fn render(&self,&ServiceSpec)->RenderedService; }
pub fn renderer_for(b: Backend) -> Option<Box<dyn ServiceRenderer>>;  // None = honest "not yet"
```

**Two design choices worth calling out:**

1. **Verbs as data.** `RenderedService` carries `enable/start/status/stop` as argv
   `Vec<String>`, *not* hidden inside an `apply()`. This makes the whole tree
   unit-testable with zero privilege ("OpenRC enables with `rc-update add holger
   default`" is an assertion, not a side effect) and lets a single privileged
   driver run them uniformly across every backend. systemd's verbs additionally
   route through the existing D-Bus `Systemd` client rather than `systemctl`.

2. **`renderer_for` returns `Option`.** Declared-but-unrendered backends resolve to
   `None` — an honest "not yet", never a faked success. This mirrors the repo's
   law (cf. the KVM `exec` "not yet" stub). `Backend::all()` + `slug()` give the
   coverage matrix its backend axis.

### B.3 The two implemented renderers

- **`SystemdRenderer { scope }`** — projects `ServiceSpec` → `SystemdServiceSpec`
  and calls the existing `systemd_unit_scoped`. `after` → `After=`; `needs` →
  `After=`+`Wants=`; user scope drops `User=`/`Group=` and targets `default.target`;
  verbs become `systemctl [--user] <verb> <name>.service`.
- **`OpenRcRenderer`** — emits `/etc/init.d/<name>` (`#!/sbin/openrc-run`,
  `supervisor="supervise-daemon"` for restart) + `/etc/conf.d/<name>` (env, mode
  0600); `depend()` carries `need net` (the `network-online.target` twin) + the
  spec's `need`/`after` edges; verbs become `rc-update`/`rc-service`.

### B.4 Why OpenRC was the chosen second backend (not systemd-user)

The deliverable offered "systemd-user *or* OpenRC". systemd-user is **nearly free**
— it is the same INI minus `User=`, already handled by the existing `Scope::User`.
Choosing it would prove little. **OpenRC is a genuinely different format** (a shell
service script, not INI; `depend()` not `After=`; `rc-update` not `systemctl`), so
it *proves the abstraction is init-neutral* rather than "systemd with a coat of
paint". The spike therefore lands **three** renderers (systemd-system, systemd-user
*for free via the bridge*, and OpenRC) — the strongest demonstration per unit of
code. Next-highest-value renderers, in order: **dinit** (richest dep model, modern,
embedded-relevant), **runit** (Void/appliances), **quadlet** (reuses systemd verbs),
then s6-rc (needs the compile step), then SysV (fallback).

### B.5 How the existing `ComponentSpec` folds in

`product::ComponentSpec` (name/binary/user/ports/after/restart/health) stays the
*airgap* unit-of-installation; it gains a `to_service_spec(&layout, backend) ->
ServiceSpec` (sibling to today's `to_systemd_spec`), so the whole AIRGAP6 install
path can pick a backend at install time. **Landed (2026-07-01):** the bridge is
`ComponentSpec::to_service_spec(&self, layout, backend)` — it wires
`working_dir` + `exec` through the `current` symlink (using the FULL `bundle_binary`
relative path, because the lifecycle path unfolds the whole product tree with
`extract_all`, not a flattened single binary), exposes the data + log dirs as
`DATA_DIR`/`LOG_DIR` env (plus the legacy `<NAME>_DATA`), maps `after` → ordering
edges and each health/listen port → a `Socket`, and picks `Scope::User` for
`Backend::SystemdUser`. `LakeSpec` is unchanged — the dep graph + start-wave
orchestration is backend-blind.

> **Layout note (2026-07-01):** `layout::Layout` gained a **`log_dir`** root
> (e.g. `/var/log`) beside `data_dir`, a `product_log(name)` → `log_dir/<name>`
> path builder, and a `prepare_product(name)` that idempotently provisions ALL
> THREE per-product dirs — `software/<version>/<name>`, `data/<name>` AND
> `log/<name>` — returning them as `(software, data, log)`. `Layout::dated` now
> takes `(data_dir, log_dir, software_dir)`. Root `prepare()` also creates the
> `log_dir` root. This is the "create data dir + create log dir" half of a product
> install (see PART E).

---

## PART C — multi-OS install test matrix + the KVM introspection probe

> **Status (2026-06-30): LANDED + LIVE.** `src/matrix.rs` (feature `matrix`) +
> the injected `probe/` crate (static-musl `skidbladnir-probe` + `skidbladnir-testsvc`)
> + the finished `src/machine.rs` KVM seam implement this whole section. A live run
> booted **4 OS guests** and the probe round-tripped a real `ProbeReport` from each:
>
> | OS | renderer | scope | result |
> |---|---|---|---|
> | Ubuntu 24.04 | systemd-system | system | **PASS** |
> | Debian 12 | systemd-system | system | **PASS** |
> | Fedora 42 | systemd-system | system | **PASS** |
> | Alpine 3.21 | OpenRC | system | **PASS** |
>
> Channels are pure-Rust + airgap-correct: the cloud-init seed is a **FAT image**
> (`fatfs`, no `genisoimage`), the payload (probe+testsvc+units) rides a second FAT
> image, and the probe reports back over an `org.skidbladnir.report` **virtio-serial**
> port (host reads a file — works under `NetMode::Isolated`). Rows sink to
> `workspace_skidbladnir/probe_reports.jsonl` + `airgap_events` (`op=matrix_probe`).
> The only external process is qemu (the hypervisor). Run:
> `cargo test --features matrix --test install_matrix_live -- --ignored --nocapture`.

### C.1 Goal

Assert, as **data**, that a real product (holger / nornir) **installs + starts +
stays healthy** across multiple OSes (⇒ multiple init backends) inside KVMs. Not
"the renderer produced a plausible string" (that is §B's unit tests) but "the
string, written into a real Alpine/Void/Debian guest, actually brought the service
up".

### C.2 The matrix axes

```
   OS image           × init backend     × scope        = a matrix cell
   ─────────────────    ───────────────    ───────────
   Ubuntu 24.04         systemd-system     system
   Fedora 41            systemd-system     system
   Debian 12            systemd / --user   system+user
   Alpine 3.20          OpenRC             system
   Void (glibc/musl)    runit              system
   Artix                OpenRC|runit|s6|dinit            (declared backends)
   Chimera              dinit              system
   (Windows Server)     SCM                machine        (out-of-band, MSI)
```

A cell is `(image, backend, scope, ServiceSpec)`. The matrix is the cartesian
product filtered to `renderer_for(backend).is_some()` — so today it runs the
systemd + OpenRC cells and **skips** (honestly, as `Skipped{reason:"no renderer"}`)
the rest, growing automatically as renderers land. This is the same honesty law as
the coverage gate in `nornir-testmatrix`.

### C.3 Reuse the existing `MachineController` (KVM + container)

`src/machine.rs` already has exactly the harness: `MachineController` trait
(`start`/`stop`/`exec`/`wait_ready`), `MachineSpec` (image, `NetMode::Isolated` for
true airgap, `Provision::CloudInit`, mounts, `Readiness`), the content-addressed
`ImageCache` (verify-on-receipt, `import_znippy_set`), `KvmController`,
`ContainerController`, and `needs_real_machine()` (→ KVM for systemd/init/airgap,
container for fast/stateless). **The matrix is built on this — no new hypervisor
code.** Per cell:

```
1. resolve the OS base image from ImageCache (znippy-carried across the gap)
2. MachineController::start(MachineSpec{ image, net: Isolated, provision: CloudInit{…} })
3. wait_ready(SystemdUnit | TcpPort)               // real init came up
4. push the airgap bundle (mount or copy) + the renderer's RenderedService files
5. run the RenderedService.enable/start verbs in-guest                (exec)
6. drop + run the PROBE (next §); collect its JSON report             (exec)
7. assert probe says Running + correct PID/port/health; stop()
```

> **Gap to close first (already tracked in `machine-control-design.md` §6):**
> `KvmController::exec` and `wait_ready(SystemdUnit)` are **deferred** — they need
> a guest channel (qemu-guest-agent QMP `guest-exec`, or an ssh key seeded via
> cloud-init). The matrix needs this. **The probe (below) is the cheapest way to
> close it**: instead of a general guest-exec, inject one static probe binary and
> have it report over a known channel, sidestepping the need for arbitrary in-guest
> exec for the assert step. Steps 4–5 still need *a* push/exec; cloud-init
> `runcmd` (rendered into the NoCloud seed ISO — also deferred but small) can run
> the install + the probe at first boot, so the matrix needs **zero** interactive
> guest exec: the guest self-installs and the probe self-reports.

### C.4 The machine-injected introspection probe

A **tiny static Rust executable** (`skidbladnir-probe`, musl static, no deps beyond
`std`) injected into each KVM. It introspects the just-installed service and emits
**one JSON line** of ground truth:

```jsonc
{
  "service": "holger", "backend": "openrc", "os": "alpine-3.20",
  "running": true, "pid": 1417,
  "ports": [{"port": 7070, "open": true}],
  "health": {"kind": "tcp", "ok": true},
  "logs_tail": ["…last N lines…"],
  "asserted_at": "2026-06-30T12:00:00Z"
}
```

How it introspects, backend-neutrally (it reads the SAME `ServiceSpec` it was
shipped with, so it knows the unit name, user, ports, health check):

- **running? / pid?** — query the active init: systemd via the `/run/systemd`
  D-Bus `ActiveState`/`MainPID` (reuse the `systemd::Systemd` client!); OpenRC via
  `rc-service <name> status` / the pidfile; runit via `sv status`; dinit via
  `dinitctl status`. The probe carries a *mini* version of `renderer_for`'s status
  knowledge. Lowest-common-denominator fallback: scan `/proc` for the exec path.
- **port open?** — a `TcpStream::connect("127.0.0.1:<port>")` from inside the guest
  (the probe is *in* the netns, so this is true even for `NetMode::Isolated`).
- **health** — run the `ComponentSpec::HealthCheck` (Tcp connect / Http GET /health).
- **logs** — tail journald (`sd-journal`) or the backend's logfile for context on
  failure.

**Report channel** (how the probe gets data back out of an airgapped guest):

1. **Primary — virtio-serial / `fw_cfg` write-back:** the probe writes its JSON to
   a virtio-serial port (`/dev/virtio-ports/org.skidbladnir.probe`) that the host
   `KvmController` reads — works with `NetMode::Isolated` (no network needed). This
   is the clean airgap-correct channel and is a *small* `KvmController` addition
   (add one `-device virtserialport` to the argv + read the host-side socket).
2. **Fallback — shared mount:** the probe writes
   `/<mount>/probe-report.jsonl`; the host reads it off the bind mount after
   `stop()`. Zero new device code (mounts already exist in `MachineSpec`).
3. **Container cells:** trivial — `ContainerController::exec` already works, so the
   probe just prints to stdout and the controller captures it.

The probe is **the introspection seam**: it converts "did the install work?" from a
human eyeballing `journalctl` into a typed `ProbeReport` row the matrix asserts on.

### C.5 Should the matrix REPORT to `workspace_skidbladnir`?

**Yes.** Reuse the existing `events::AirgapEvent` / `JsonlSink` (already
"persist-to-warehouse" by design — one row per pack/fetch/unfold/install/start
boundary). Add an `op::MATRIX_PROBE` event carrying the `ProbeReport` per cell, sunk
to `<workspace_skidbladnir>/airgap_events.jsonl`. nornir's autonom completeness gate
(`nornir/src/autonom.rs`, the `surface_coverage` Iceberg table) can then ingest the
matrix the same way it ingests other surfaces — a `(backend × os)` install-coverage
axis: green ⟺ every implemented backend has a passing install+probe on at least one
OS. This makes the install tree's completeness a **mechanically-checked datum**, not
a claim, exactly matching the nordisk functional-matrix philosophy. Concretely: the
matrix writes `ProbeReport` rows to `workspace_skidbladnir`; nornir's testmatrix
reads them as an `InstallCoverage` surface. Skíðblaðnir stays the *producer*; nornir
stays the *verdict engine*.

### C.6 Build order for Part C

1. `ProbeReport` type + `skidbladnir-probe` bin (static, reads `ServiceSpec`,
   `/proc` + TCP introspection) — pure, unit-testable on the host first.
2. virtio-serial write-back in `KvmController` (+ cloud-init seed-ISO render — the
   two deferred `machine.rs` items, both small).
3. The `Matrix` driver: cartesian `(image × backend)`, `renderer_for` filter,
   per-cell `MachineController` run, collect `ProbeReport`s.
4. Sink `ProbeReport` → `AirgapEvent` → `workspace_skidbladnir`; nornir ingests.

---

## PART D — tunnr: part of Skíðblaðnir, or a cooperating sibling?

### D.1 What tunnr is (from the references — the repo is not yet checked out here)

tunnr is Rickard's invention for **thin, OS-less laptops/desktops**: a pure-Rust
**UEFI appliance** (its "binary" in the airgap bundle is a **disk image**, not an
ELF). The references are consistent: `product::ComponentSpec::tunnr()` declares it
as `run --appliance`, user `_tunnr`, HTTPS admin on 8443, HTTP `/healthz`; the
install-system doc calls it `type: Appliance` (vs `Service`), writes the image to a
block device / loop file, and registers it with a **tunnr KVM supervisor** (itself a
service that manages VMs). It is the "OS-less client" endpoint of the stack — a
machine that boots straight into the appliance image with no general-purpose OS.

### D.2 Recommendation — **cooperating sibling, composed by Skíðblaðnir; NOT absorbed**

tunnr should remain its **own leaf repo**, a sibling of Skíðblaðnir, for the same
reason Skíðblaðnir was extracted out of nornir (the Cargo.toml comment says it
plainly: "so leaf consumers — nornir, holger, **tunnr** — can pull the airgap
framework without a release-cascade"). Folding tunnr *into* Skíðblaðnir would
re-couple the very things that extraction decoupled, and would drag a UEFI/appliance
image-build toolchain into the lean airgap core. They compose cleanly **without**
merging:

- **Skíðblaðnir is the mover; tunnr is one kind of cargo.** tunnr is *described* by
  a spec (`ComponentSpec::tunnr()` already exists) and *carried* by the bundle —
  exactly like holger, which is also a sibling, not a dependency. Skíðblaðnir never
  links tunnr's code; it ships its image and renders its install.
- **tunnr is an `Appliance`, which the tree should model as a first-class kind.**
  Add `ServiceSpec`'s sibling for non-service installs: an `ApplianceSpec` (write
  image to block dev / loop, register with the tunnr KVM supervisor, health via
  `tunnr status <vm> == BOOT-OK`). The install tree then has two arms — **service**
  (this doc, all the init backends) and **appliance** (tunnr's image-to-metal). The
  `MachineController`/`ImageCache` that Part C uses to *test* installs is the SAME
  machinery tunnr's supervisor uses to *run* appliances — so they share the KVM/image
  layer without sharing a crate.
- **The dependency direction is one-way and clean:** tunnr → depends on →
  Skíðblaðnir (pulls the airgap framework + the install tree + `MachineController`).
  Skíðblaðnir → only *describes* tunnr (a registry spec). No cycle.
- **Why not absorb, concretely:** (1) release cadence — an appliance image format
  changes on a different clock than the airgap engine; (2) leanness — the embedded
  Skíðblaðnir client must not pull a UEFI image builder; (3) blast radius — tunnr is
  an *endpoint product*, Skíðblaðnir is *infrastructure* every product depends on.

**Verdict:** keep tunnr a sibling. Have Skíðblaðnir grow an `ApplianceSpec` arm
(parallel to `ServiceSpec`) and let tunnr consume Skíðblaðnir's install tree +
machine layer. They cooperate; they do not merge. When the tunnr repo lands, the
first concrete step is moving `ComponentSpec::tunnr()`'s appliance knowledge into an
`ApplianceSpec` that tunnr owns and Skíðblaðnir renders — the same
"product-owns-its-spec, engine-applies-it" split holger already follows.

---

## PART E — product install lifecycle (install vs update vs uninstall; data + log provisioning)

> **Status (2026-07-01): LANDED.** `src/install.rs` turns install from a
> binary-swap into a first-class **product** lifecycle over the neutral service
> tree (PART B) + the AIRGAP5 layout (now with a per-product LOG dir, see the B.5
> layout note). Every op is **verbs-as-data**: the lifecycle fn does the
> UNPRIVILEGED filesystem work (dirs, unfold, unit files under a fs `root`) and
> returns the privileged service-manager argv as data — a thin `apply()` runs them.
> That split makes the whole lifecycle unit-testable offline into a tempdir.

### E.1 The three verbs

Rickard's framing — install ≠ update ≠ uninstall:

- **install** = (1) create the service, (2) unpack the airgapped product from its
  znippy archive, (3) create the data dir + create the log dir.
- **update**  = swap exe / new generation, **preserve** data + log, restart.
- **uninstall** = delete the service; optionally purge data + log.

### E.2 API (`src/install.rs`)

```rust
pub enum InstallOp { Install, Update, Uninstall }

pub struct InstallOutcome {
    pub op: InstallOp,
    pub product: String,
    pub backend: Backend,
    pub version: String,
    pub dirs: Vec<PathBuf>,          // created/ensured: software/<ver>/<name>, data/<name>, log/<name>
    pub removed: Vec<PathBuf>,       // purged data/log + deleted unit files (uninstall)
    pub files_written: Vec<PathBuf>, // unit/script/env files (under the fs `root`)
    pub identity: Option<String>,    // minted (install) / preserved (update) wire string
    pub rendered: RenderedService,   // file bodies + enable/start/status/stop verbs
    pub verbs: Vec<Vec<String>>,     // the ORDERED privileged argv to run for this op
}

pub fn install_product(spec, layout, bundle, archive, backend, root) -> Result<InstallOutcome>;
pub fn update_product (spec, layout, bundle, archive, backend, root) -> Result<InstallOutcome>;
pub fn uninstall_product(spec, layout, backend, root, purge: bool)   -> Result<InstallOutcome>;
pub fn apply(&InstallOutcome) -> Result<()>;   // the thin privileged driver (runs `verbs`)
```

- **`root`** is the filesystem root the absolute unit paths
  (`/etc/systemd/system/…`, `/etc/init.d/…`) are re-rooted under — `/` in
  production, a tempdir in tests, so writes are assertable without privilege.
- **`install_product`**: `layout.prepare_product` (software+data+log) →
  `archive.extract_all` (unfold the WHOLE product) → `+x` the binary →
  `layout.activate()` (atomic `current` swap) → `identity::ensure` under the data
  dir → render for `backend` → write unit files → `verbs = [enable, start]`.
- **`update_product`**: same, but `layout` carries the NEW `version`;
  `prepare_product` is idempotent so an existing `data/<name>`+`log/<name>` is
  **kept, never clobbered**, and `identity::ensure` LOADS the existing id (the
  reinstall-stable install id survives). Old generation retained for rollback.
  `verbs = [stop, start]` (restart; the unit points through `current`, so it never
  churns → no daemon-reload).
- **`uninstall_product`**: render → remove the unit/script/env files → if `purge`,
  `remove_dir_all` the `data/<name>`+`log/<name>` (else retain) → `verbs = [stop,
  disable]`. Never touches the shared software generation.

### E.3 Seams (honest)

- **No native `StateDirectory=`/`LogsDirectory=`.** `SystemdServiceSpec` has no such
  field, so the data + log dirs are surfaced as `Environment=DATA_DIR/LOG_DIR` +
  `WorkingDirectory=` rather than the native systemd directives; the dirs are
  provisioned explicitly by `prepare_product`, so nothing is lost — but a future
  pass could add real `StateDirectory=`/`LogsDirectory=` on the systemd renderer.
- **`extract_all` vs `extract_one`.** The lifecycle unfolds the whole product tree
  (`extract_all`, tree preserved), so `to_service_spec`'s exec is the full
  `bundle_binary` relative path under `current/<name>`. The legacy AIRGAP6
  `SystemdInstaller`/`to_systemd_spec` path FLATTENS via `extract_one` to
  `current/<name>/<binfile>` (`installed_binary`). Both are correct for their own
  unpack; they are deliberately not merged here.
- **No firewall step.** `install_product` does not open ports (the AIRGAP6
  `SystemdInstaller` did); ports are declared as `Socket`s on the neutral spec and
  can be opened by the same `open_ports` helper when a privileged driver wires it.

### E.4 Tests (`src/install.rs`, offline into tempdirs)

- `install_product_on_systemd_system`, `install_product_on_openrc` — full
  inject-assert: all three dirs exist, whole product unfolded + binary `+x`,
  `identity.toml` under data, `current` resolves, unit files reference the data +
  log + exec paths, verbs `= [enable, start]` (OpenRC: `rc-update add` + init.d /
  conf.d files).
- `update_product_preserves_data_and_log_and_swaps_version` — writes real state
  into data + log, updates to a new gen, asserts both survive, install id
  preserved, `active_version` swapped, old gen retained, verbs `= [stop, start]`.
- `uninstall_product_purge_flag_controls_data_retention` — `purge=false` keeps
  data+log and removes the unit (`verbs = [stop, disable]`); `purge=true` removes
  data+log too.
- `install_outcome_json_roundtrips` — the outcome is plain serde data.
- Layout: `prepare_product_creates_software_data_and_log_dirs` (idempotent, never
  clobbers). Bridge: `component_spec_renders_neutral_service_spec_with_data_and_log`.

---

## Appendix — spike status

- **Branch:** `agent/install-tree-spike` (worktree `/home/rickard/git/install-tree-spike-wt`). **Not pushed.**
- **Landed:** `src/service.rs` (the tree) + a one-line `lib.rs` module hook.
- **Renderers:** systemd-system, systemd-user (both via the existing
  `systemd_unit_scoped` bridge), OpenRC (native).
- **Tests:** 6 inject-assert tests incl. `one_spec_renders_to_systemd_and_openrc`
  (the headline: ONE `ServiceSpec` → both backends, each in its native shape).
  `cargo test` green (82 passed, 1 ignored), `cargo clippy` clean.
- **Not done (design only):** the other 5 init renderers, the `apply()` privileged
  driver, `ComponentSpec::to_service_spec`, the matrix driver, the probe binary,
  the virtio-serial/cloud-init `machine.rs` additions, the `ApplianceSpec` arm.
