# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What This Project Is

sdme boots systemd-nspawn containers using overlayfs copy-on-write layers, with OCI registry integration and Kubernetes Pod YAML support for local development and testing. It produces a single binary `sdme` that manages containers from explicit root filesystems, keeping the base rootfs untouched.

Runs on Linux with systemd. Requires root for all operations. Uses kernel overlayfs for copy-on-write storage. By default, containers are overlayfs clones of `/`. Also supports importing rootfs from other distros (Ubuntu, Debian, Fedora, NixOS). Imported rootfs needs systemd and dbus.

## Build & Test

```bash
cargo build --release       # build the binary (with embedded probe)
cargo test                  # run all tests
cargo test <test_name>      # run a single test
make                        # same as cargo build --release
make deb                    # build .deb package
make rpm                    # build .rpm package
make pkg                    # build .pkg.tar.zst package (Arch Linux)
sudo make install           # install to /usr/local (does NOT rebuild)
```

The probe binary (`sdme-kube-probe`) is automatically built and embedded into sdme
by the `build.rs` script. A single `cargo build --release` handles everything.
The inner probe build uses a separate target directory (`target/probe-build/`) to
avoid cargo lock contention. Override with `SDME_KUBE_PROBE_PATH` env var or skip
with `SDME_SKIP_PROBE_BUILD=1` (probe-less build).

### Release

Static musl binaries (x86_64 + aarch64) are built with `cargo-zigbuild`. Locally:

```bash
./dist/build-release.sh            # build all targets to target/dist/
./dist/build-release.sh -v <target> # build one target, verbose
```

CI: pushing a `v*` tag triggers `.github/workflows/release.yml`, which runs tests, cross-compiles both targets, generates SHA256SUMS, and creates a GitHub release with all artifacts.

## Architecture

The project is a single Rust binary (`src/main.rs`) backed by a shared library (`src/lib.rs`). CLI parsing uses clap with derive.

### Core Concepts

- **Overlayfs CoW storage**: each container gets `upper/work/merged` directories under the datadir. The lower layer is the imported rootfs. Uses kernel overlayfs.
- **Systemd integration**: containers are managed as a systemd template unit (`sdme@.service`) with `Type=notify`. Start goes through D-Bus to systemd; the unit is considered active only after `systemd-nspawn` sends `sd_notify(READY=1)`. The systemd `TimeoutStartSec` is set to `boot_timeout + 30` seconds so the Rust-side wait loop always expires before systemd kills the container. Exit code 133 is treated as a successful clean shutdown (`SuccessExitStatus=133`, `RestartForceExitStatus=133`). The template unit is auto-installed and auto-updated when content changes. `TasksMax` is configurable via the `tasks_max` config key (default 16384).
- **machinectl integration**: `join` and `exec` use `machinectl shell` for container interaction. `stop` uses D-Bus (`KillMachine` for graceful/kill, `TerminateMachine` for terminate).
- **DNS resolution**: containers share the host's network namespace. By default, `systemd-resolved` is masked in the overlayfs upper layer at creation time so the host's resolver handles DNS. The `default_create_masked_services` config key (default: `"systemd-resolved.service"`) controls which services are masked; the `--masked-services` CLI flag overrides per container. When using `--network-zone` with config defaults, `systemd-resolved.service` is automatically removed from the mask list so LLMNR/mDNS enables inter-container name resolution. For zone containers where resolved is not masked: `/etc/resolv.conf` is symlinked to the resolved stub (`../run/systemd/resolve/stub-resolv.conf`) and an LLMNR/mDNS drop-in (`resolved.conf.d/zone-llmnr.conf`) is written. For non-zone containers, a placeholder `/etc/resolv.conf` regular file is written so `systemd-nspawn --resolv-conf=auto` can populate it at boot. NixOS rootfs always skip masking. When `--network-veth`, `--network-zone`, or `--network-bridge` is used, `systemd-networkd` is auto-enabled in the overlayfs upper layer so the container-side interface (`host0`) gets an IP via DHCP. **NixOS limitation**: the default NixOS config has `networkd` enabled but `resolved` disabled; `--network-veth` and `--network-bridge` work out of the box, but `--network-zone` needs resolved for LLMNR/mDNS — add `services.resolved.enable = true` to the nix expression to enable it.
- **State files**: container metadata persisted as KEY=VALUE files under `{datadir}/state/{name}`. Written atomically with 0o600 permissions via `atomic_write_mode()`. OCI env files are also 0o600 (may contain secrets); ports and volumes files are 0o644.
- **Health checks**: `sdme ps` detects broken containers (missing dirs, missing rootfs) and reports health status with OS detection via os-release.
### CLI Commands

| Command | Description |
|---------|-------------|
| `sdme new` | Create, start, and enter a new container (accepts same flags as `create`) |
| `sdme cp` | Copy files between host, containers, and root filesystems |
| `sdme create` | Create a new container (overlayfs dirs + state file). Security flags: `--strict`, `--hardened`, `--drop-capability`, `--capability`, `--no-new-privileges`, `--read-only`, `--system-call-filter`, `--apparmor-profile`. OCI flags: `--no-oci-ports`, `--no-oci-volumes`, `--oci-env KEY=VALUE` (sets env vars for the OCI app service via the `/oci/apps/{name}/env` file, separate from `-e` which sets nspawn env vars). `--masked-services` overrides the `default_create_masked_services` config (comma-separated systemd units to mask) |
| `sdme start` | Start one or more containers (installs/updates template unit, starts via D-Bus). Supports `--all` to start all stopped containers |
| `sdme join` | Enter a running container (`machinectl shell`). `--start` starts the container first if stopped. `--oci [APP]` enters the OCI app's PID/IPC/mount namespaces via `nsenter` (default shell: `/bin/sh`; optional app name for multi-container kube pods) |
| `sdme exec` | Run a one-off command in a running container (`machinectl shell`). `--oci [APP]` enters the OCI app's PID, IPC, and mount namespaces via `nsenter` (discovers the app's host PID from its cgroup, then runs `nsenter -t <pid> --pid --ipc --mount`; optional app name for multi-container kube pods) |
| `sdme stop` | Graceful shutdown via `SIGRTMIN+4` (default), `--term` for terminate, `--kill` for force-kill |
| `sdme rm` | Remove containers (stops if running, deletes state + files) |
| `sdme ps` | List containers with status, health, OS, pod/OCI-pod/kube/userns/binds (if any) |
| `sdme logs` | View container logs (exec's `journalctl`). `--oci [APP]` shows the OCI app service logs (`journalctl -u sdme-oci-{name}.service` inside the container; optional app name for multi-container kube pods) |
| `sdme fs import` | Import a rootfs from a directory, tarball, URL, OCI image, or QCOW2 disk image |
| `sdme fs ls` | List imported root filesystems |
| `sdme fs rm` | Remove imported root filesystems |
| `sdme fs build` | Build a root filesystem from a build config. Resumes from failed step on re-run if config unchanged; `--no-cache` forces clean build |
| `sdme fs export` | Export a container (default) or rootfs (`fs:` prefix) to a directory, tarball, or raw disk image (`--filesystem ext4\|btrfs`). `--timezone` sets the timezone in the exported rootfs. Holds shared flock to prevent `fs rm`/`sdme rm` during export |
| `sdme fs cache info/ls/clean` | Manage the OCI blob cache (info, list, clean `[--all]`) |
| `sdme fs gc` | Clean up stale transaction artifacts from interrupted operations |
| `sdme set` | Set resource limits on a container (replaces all limits) |
| `sdme config get/set` | View or modify configuration |
| `sdme pod new` | Create a new pod (shared network namespace) |
| `sdme pod ls` | List pods |
| `sdme pod rm` | Remove pods |
| `sdme kube apply` | Create, start, and enter a container from a Kubernetes Pod YAML (`-f file`, `--base-fs` required) |
| `sdme kube create` | Create a container from a Kubernetes Pod YAML (no start) |
| `sdme kube delete` | Stop and remove a kube container and its rootfs (`--force` allows non-kube containers) |
| `sdme kube secret create/ls/rm` | Manage secrets for kube pods |
| `sdme kube configmap create/ls/rm` | Manage configmaps for kube pods |
| `sdme enable` | Enable containers to auto-start on boot |
| `sdme disable` | Disable container auto-start |
| `sdme config apparmor-profile` | Print the default AppArmor profile for sdme containers |
| `sdme config completions` | Generate shell completions (Bash, Fish, Zsh) |

**CLI help is the canonical documentation.** There is no manpage. Each command's `--help` includes man-page-style reference sections (examples, environment, files, exit status, notes). The help text lives in `*_HELP` constants at the top of `src/main.rs` and is wired to clap via `after_long_help`. When adding or changing features, update the relevant `*_HELP` constant so `--help` stays in sync.

### dist/ Directory

The `dist/` directory contains both checked-in packaging files and generated build outputs:

- `dist/deb/postinst`: Debian post-install script (checked in)
- `dist/out/completions/`: shell completions (generated by `make dist/out/completions`, gitignored)
- `dist/out/apparmor/`: AppArmor profile (generated in CI, gitignored)

### Key Modules

| File | Purpose |
|------|---------|
| `src/main.rs` | CLI entry point (clap derive), command dispatch |
| `src/lib.rs` | Shared types: `State` (KEY=VALUE), `validate_name`, `sudo_user`, global interrupt handler (`INTERRUPTED`, `check_interrupted`, `install_interrupt_handler` for SIGINT+SIGTERM) |
| `src/containers.rs` | Container create/remove/join/exec/stop/list, overlayfs directory management, overlay mount/unmount helpers, DNS setup, volume directory management |
| `src/cp.rs` | File copy between host, containers, and rootfs (`sdme cp`) |
| `src/systemd.rs` | D-Bus helpers (start/status/stop), template unit generation (`Type=notify`), env files, boot/shutdown waiting |
| `src/system_check.rs` | Version checks (systemd), dependency checks (`find_program`) |
| `src/rootfs.rs` | Rootfs listing, removal, os-release parsing, distro detection |
| `src/export.rs` | Rootfs export (dir copy, tarball creation, raw ext4/btrfs disk image) |
| `src/import/mod.rs` | Rootfs import orchestration: source detection, URL download (with proxy support), systemd detection |
| `src/import/dir.rs` | Directory-based rootfs import |
| `src/import/tar.rs` | Tarball extraction with magic-byte compression detection |
| `src/import/img.rs` | QCOW2 and raw disk image import via qemu-nbd |
| `src/oci/mod.rs` | OCI module index, app name derivation from registry references, sorted-keys helper |
| `src/oci/app.rs` | OCI app setup: user resolution, systemd service unit generation, isolate binary deployment |
| `src/oci/cache.rs` | Content-addressable OCI blob cache with LRU eviction |
| `src/oci/layout.rs` | OCI tarball detection, manifest resolution, layer unpacking with whiteout handling |
| `src/oci/registry.rs` | OCI registry pulling via Distribution Spec (docker.io, quay.io, etc.) |
| `src/oci/rootfs.rs` | OCI rootfs helpers: app name detection, port/volume reading from OCI metadata files |
| `src/names.rs` | Container name generation from a Tupi-Guarani wordlist with collision avoidance |
| `src/config.rs` | Config file loading/saving (`/etc/sdme.conf`) |
| `src/build.rs` | Build config parsing and rootfs build execution |
| `src/lock.rs` | Advisory file locking (`flock`) for resource protection across all mutating operations |
| `src/txn.rs` | Enumerated transaction staging (`.{name}.{kind}-txn-{pid}`) and `sdme fs gc` helpers |
| `src/copy.rs` | Filesystem tree copying with hard link preservation, xattr support, special file handling, path sanitization |
| `src/mounts.rs` | Bind mount (`BindConfig`) and environment variable (`EnvConfig`) configuration |
| `src/network.rs` | Network configuration validation and state serialization |
| `src/security.rs` | Security hardening: `SecurityConfig` (capabilities, seccomp, no-new-privileges, read-only, AppArmor), state file roundtrip, nspawn arg generation, validation |
| `src/kube/` | Kubernetes Pod YAML support: types, plan validation, container creation, kube delete, shared store abstraction for secrets and configmaps |
| `src/kube/probe/` | Embedded `sdme-kube-probe` binary: CLI entry point, probe runner (failure counting/actions), exec/http/tcp/grpc check implementations |
| `src/pod.rs` | Pod (shared network namespace) lifecycle: create, list, remove, runtime netns management |
| `src/elf.rs` | Shared `Arch` enum and minimal ELF64 header builder for static binaries (used by isolate and devfd_shim) |
| `src/isolate/` | Static ELF binary generation for PID/IPC namespace isolation in OCI app services |
| `src/devfd_shim/` | LD_PRELOAD shim generation for `/dev/fd/` interception in containers |

### Rust Dependencies

- `clap`: CLI parsing (derive)
- `zbus`: D-Bus communication with systemd (blocking API)
- `libc`: syscalls for rootfs import (lchown, mknod, etc.), privilege dropping
- `anyhow`: error handling
- `serde`/`toml`: config file parsing
- `tar`: archive extraction with xattr support
- `flate2`: gzip decompression
- `bzip2`: bzip2 decompression
- `xz2`: xz/lzma decompression
- `zstd`: zstd decompression
- `serde_json`: JSON parsing (OCI image manifests)
- `ureq`: HTTP client for URL downloads and OCI registry pulling (blocking, rustls TLS)
- `sha2`: SHA-256 hashing (OCI digest verification)
- `serde_yml`: YAML parsing (Kubernetes Pod manifests)
- `tonic`: gRPC client for probe binary (optional, `probe` feature)
- `prost`: protobuf serialization for gRPC health checks (optional, `probe` feature)
- `tokio`: async runtime for gRPC probes (optional, `probe` feature)
- `clap_complete`: shell completion generation (Bash, Fish, Zsh)

### External Dependencies

| Program | Package | Required for |
|---------|---------|--------------|
| `systemd` (>= 252) | `systemd` | All commands (D-Bus communication) |
| `systemd-nspawn` | `systemd-container` | Running containers |
| `machinectl` | `systemd-container` | `sdme join`, `sdme exec`, `sdme new` |
| `busctl` | `systemd` | Boot-wait D-Bus probe for `--userns` containers |
| `journalctl` | `systemd` | `sdme logs` |
| `nsenter` | `util-linux` | `sdme exec --oci`, `sdme join --oci` (namespace entry) |
| `sfdisk` | `util-linux` | `sdme fs export --vm` (GPT partition table) |
| `mkswap` | `util-linux` | `sdme fs export --vm --swap` (swap partition) |
| `qemu-nbd` | `qemu-utils` | `sdme fs import` (QCOW2 images only) |
| `mkfs.ext4` | `e2fsprogs` | `sdme fs export` (ext4 raw images, default) |
| `mkfs.btrfs` | `btrfs-progs` | `sdme fs export` (btrfs raw images only) |

Dependencies are checked at runtime before use via `system_check::check_dependencies()`, which resolves each binary in PATH and prints the resolved path with `-v`.

## Design Decisions

- **Root-only**: sdme requires root (`euid == 0`). Checked at program start.
- **Datadir**: always `/var/lib/sdme`.
- **Container management**: `join` and `exec` spawn `machinectl shell` and forward the exit status. `stop` has three tiers: graceful (default; `KillMachine` SIGRTMIN+4 to leader), terminate (`--term`; `TerminateMachine`), force-kill (`--kill`; `KillMachine` SIGKILL to all). `--term` and `--kill` are mutually exclusive. Timeouts are configurable via `stop_timeout_graceful` (default 90s), `stop_timeout_terminate` (default 30s), and `stop_timeout_kill` (default 15s) config keys. Internal callers use `StopMode::Terminate` with a hardcoded 30s timeout.
- **D-Bus**: used for `start_unit`, `daemon_reload`, `is_unit_active`, `get_systemd_version`, `kill_machine`, `terminate_machine`. Always system bus.
- **Rootfs import sources**: `sdme fs import` auto-detects source type: URL, directory, QCOW2 (via `qemu-nbd`), OCI tarball (detected by `oci-layout` file), OCI registry reference, or plain tarball (compression detected from magic bytes). After import, systemd is detected; if missing, distro-specific packages are installed via chroot (`--install-packages` controls this). Supported distro families: Debian (apt-get), Fedora (dnf), Arch (pacman), NixOS (no-op, already has systemd).
- **NixOS rootfs**: Pre-built NixOS rootfs (recognized by `ID=nixos` in os-release) are supported as containers. NixOS rootfs must be built externally and imported as a directory; `test/scripts/build-nixos-rootfs.sh` and `test/nix/sdme-nixos.nix` provide an example pipeline that builds NixOS via `docker.io/nixos/nix` and `nix-build`. `patch_rootfs_services` is skipped for NixOS rootfs because NixOS manages `/etc/systemd/system` as an immutable symlink to the Nix store via activation; creating files there would prevent activation from setting up systemd units. **Network limitation**: `networkd` is enabled but `resolved` is disabled by default. `--network-veth` and `--network-bridge` get DHCP via networkd and work out of the box. `--network-zone` also needs resolved for LLMNR/mDNS inter-container name resolution; add `services.resolved.enable = true` to the nix expression to enable it.
- **HTTP proxy support**: URL downloads respect standard proxy env vars (`https_proxy`, `http_proxy`, `all_proxy`, and uppercase variants). Since sdme runs as root, users must pass proxy variables through sudo (e.g. `sudo -E`). Configured in `build_http_agent()` (`src/import/mod.rs`). HTTP timeouts are configurable via `http_timeout` (connect/resolve, default 30s) and `http_body_timeout` (body receive, default 300s). Download size is capped by `max_download_size` (default 50G, `0` = unlimited).
- **Rootfs patching at import**: patches imported rootfs for nspawn compatibility: unmasks `systemd-logind` if masked, removes pre-existing `systemd-resolved.service -> /dev/null` masks from the rootfs (so they don't leak through overlayfs and interfere with zone containers that need resolved), and installs missing packages for `machinectl shell` (e.g. `util-linux`, `pam` on RHEL-family). Skipped for `DistroFamily::NixOS` (NixOS manages `/etc` via activation). Service masking (e.g. `systemd-resolved`) is handled at container create time, not import time, via the configurable `--masked-services` / `default_create_masked_services` mechanism.
- **Configurable distro prehooks**: Import and export distro-specific chroot commands can be overridden via `[distros.<family>]` sections in `/etc/sdme.conf`. Three hooks per distro: `import_prehook` (commands to make a rootfs bootable under nspawn: systemd, dbus, pam/login, tzdata), `export_prehook` (commands for container/rootfs exports; empty builtins today), and `export_vm_prehook` (commands to prepare a rootfs for VM export via `fs export --vm`: udev install). Absent = use built-in defaults. Empty array = explicitly do nothing. Config key names match `DistroFamily::config_key()`: `debian`, `fedora`, `arch`, `suse`, `nixos`, `unknown`. CLI: `sdme config set distros.debian.import_prehook '["cmd1","cmd2"]'`, clear with empty string. Detection logic (is systemd/pam/udev present?) stays in Rust code; hooks define *what* to run.
- **Opaque dirs**: `-o` / `--overlayfs-opaque-dirs` on `create`/`new` sets `trusted.overlay.opaque` xattr, hiding lower-layer contents. For host-rootfs containers, `host_rootfs_opaque_dirs` config applies when no `-o` given (default: `/etc/systemd/system,/var/log`). Paths validated by `containers::validate_opaque_dirs()` (absolute, no `..`, no duplicates). Merge logic in `resolve_opaque_dirs()` in `main.rs`. When `/etc/systemd/system` is opaque, the `dbus.service` symlink (alias for the D-Bus implementation, e.g. `dbus-broker.service`) is preserved from the lower layer into the upper layer so `dbus.socket` can activate its service.
- **Umask check**: `containers::create()` refuses to proceed when the process umask strips read or execute from "other" (`umask & 005 != 0`). A restrictive umask causes files in the overlayfs upper layer to be inaccessible to non-root services (e.g. dbus-daemon as `messagebus`), preventing boot.
- **Bind mounts and env vars**: `-b`/`--bind` and `-e`/`--env` on `create`/`new` add custom bind mounts and environment variables. Stored in the state file and converted to systemd-nspawn flags at start time. Bind mounts validated (absolute paths, no `..`). Managed by `BindConfig` and `EnvConfig` in `src/mounts.rs`.
- **OCI port forwarding**: OCI images declare exposed ports in `/oci/apps/{name}/ports` (written at import time). On `create`/`new`, `read_oci_ports()` reads the file and `auto_wire_oci_ports()` (in `main.rs`) merges them into the network config as `--port` rules when the container has a private network namespace. User `--port` flags take priority (matching container ports are skipped). On host-network containers, an informational message is printed instead. Suppressed with `--no-oci-ports`.
- **OCI volume mounts**: OCI images declare volumes in `/oci/apps/{name}/volumes` (written at import time). On `create`/`new`, `read_oci_volumes()` reads the file and `do_create()` creates host-side directories at `{datadir}/volumes/{container}/{volume-name}` with bind mounts to `/oci/apps/{name}/root{volume-path}`. User `--bind` flags take priority (matching container paths are skipped). Volume data survives container removal (`sdme rm` prints the path but does not delete it). Suppressed with `--no-oci-volumes`. Stored as `OCI_VOLUMES` in the state file.
- **OCI registry pulling**: supports pulling from OCI registries (e.g. `docker.io/ubuntu:24.04`). Implements the OCI Distribution Spec in `src/oci/registry.rs`; resolves tags to manifests, matches architecture, downloads and extracts layers. Resolved manifests are cached locally under `{cache_dir}/manifests/` with a configurable TTL (`oci_manifest_cache_ttl`, default 900s / 15 minutes, `0` disables). The `--no-cache` flag on `fs import`, `kube apply`, and `kube create` overrides the TTL to 0 for a single invocation, forcing a fresh manifest fetch. Supports `--oci-mode` and `--base-fs` for running OCI app images as systemd services. The `default_base_fs` config key provides a default `--base-fs` value for OCI app imports when the flag is not specified on the command line. OCI app files are placed under `/oci/apps/{name}/` (root, env, ports, volumes). The app name is derived from the last path component of the registry repository (underscores replaced with hyphens) for registry images, or the rootfs name for non-registry imports. Stored as `OCI_APP={name}` in the container state file.
- **NixOS OCI unit placement**: NixOS activation replaces `/etc/systemd/system` with an immutable symlink to the Nix store, which would destroy OCI app unit files written to the overlayfs upper layer. On NixOS, OCI app units (services, timers, drop-ins) are placed in `/etc/systemd/system.control/` instead; this is the highest-priority persistent unit search path and is not managed by NixOS activation. Detection uses `detect_distro_family()` (os-release `ID=nixos`). The `systemd_unit_dir()` function in `src/oci/app.rs` returns the appropriate relative path. All call sites (OCI app setup, kube volume service, pod drop-ins, hardening drop-ins, service detection in `main.rs`) use this function or check both directories.
- **Pods**: `sdme pod new` creates a shared network namespace (loopback only) that multiple containers can join. The pod netns is created with `unshare(CLONE_NEWNET)` and bind-mounted to `/run/sdme/pods/{name}/netns`. Persistent state lives at `{datadir}/pods/{name}/state`. Two join mechanisms: `--pod` puts the entire nspawn container in the pod's netns via `--network-namespace-path=` (incompatible with `--userns`/`--hardened` because the kernel blocks `setns(CLONE_NEWNET)` across user namespace boundaries; `--private-network` is automatically omitted since the pod's netns provides equivalent loopback-only isolation); `--oci-pod` bind-mounts the pod's netns into the container and uses an inner systemd drop-in (`NetworkNamespacePath=`) so only the OCI app service process enters the pod's netns. `--oci-pod` requires `--private-network` (or `--hardened`/`--strict` which imply it) because systemd-nspawn strips `CAP_NET_ADMIN` on host-network containers and the inner systemd refuses `NetworkNamespacePath=` without that capability. This works with `--hardened` since the netns join happens inside the container's own network namespace (requires an OCI app rootfs with an `sdme-oci-{name}.service` unit). Both flags can be combined on the same container. Container state stores `POD` and/or `OCI_POD` keys. Pod removal checks both keys.
- **User namespace isolation**: `-u`/`--userns` on `create`/`new` enables user namespace isolation via `--private-users=pick --private-users-ownership=auto`. Container root maps to a high UID on the host (524288+ range, deterministically hashed from machine name). On kernel 6.6+, overlayfs supports idmapped mounts (zero overhead, files stay UID 0 on disk). Stored as `USERNS=yes` in the container state file. Boot-wait D-Bus probing uses `busctl --machine=` for userns containers because (a) the kernel blocks `/proc/{leader}/root/` traversal across user namespace boundaries, and (b) in-process `setns(CLONE_NEWUSER)` requires a single-threaded caller but zbus has already spawned threads. `busctl` handles this internally by forking a helper child. Detection is by comparing inode numbers of `/proc/self/ns/user` vs `/proc/{leader}/ns/user`; standard containers use the direct zbus path unchanged. **Limitation**: `--private-users-ownership=auto` uses kernel idmapped mounts, which fail if the rootfs contains files with `security.capability` xattrs (e.g. openSUSE Tumbleweed ships `newuidmap`/`newgidmap` with file capabilities instead of setuid bits). The kernel refuses to create an idmapped mount when `security.capability` xattrs are present because they cannot be safely remapped across user namespace boundaries. The built-in Suse import prehook strips them automatically; both export prehooks restore them. For other affected distros, custom `import_prehook` and `export_vm_prehook` overrides can be configured.
- **Security hardening**: Three tiers of security: (1) Individual flags: `--drop-capability` / `--capability` (add/drop Linux capabilities, validated against `KNOWN_CAPS`, accepts with/without `CAP_` prefix), `--no-new-privileges` (blocks privilege escalation via setuid/file capabilities), `--read-only` (mounts rootfs read-only), `--system-call-filter` (seccomp filters, `@group` or `~@group` syntax), `--apparmor-profile` (applied as `AppArmorProfile=` in the systemd unit drop-in, not as an nspawn flag). (2) `--hardened`: enables user namespace isolation, private network, `--no-new-privileges`, and drops capabilities from the `hardened_drop_caps` config key (default: `CAP_SYS_PTRACE,CAP_NET_RAW,CAP_SYS_RAWIO,CAP_SYS_BOOT`). (3) `--strict`: implies `--hardened` plus Docker-equivalent cap drops (retains only Docker's ~14 caps + `CAP_SYS_ADMIN` for systemd), seccomp filters (`~@cpu-emulation,~@debug,~@obsolete,~@raw-io`), and the `sdme-default` AppArmor profile. Constants in `security.rs`: `HARDENED_DROP_CAPS`, `STRICT_DROP_CAPS`, `STRICT_SYSCALL_FILTERS`, `STRICT_APPARMOR_PROFILE`, `APPARMOR_PROFILE` (the profile text). All security options are stored in the state file (`DROP_CAPS`, `ADD_CAPS`, `NO_NEW_PRIVS`, `READ_ONLY`, `SYSCALL_FILTER`, `APPARMOR_PROFILE`) and read back at start time. Managed by `SecurityConfig` in `src/security.rs`. `sdme config apparmor-profile` dumps the default AppArmor profile to stdout for installation.
- **Interrupt handling**: a global `INTERRUPTED` flag (`src/lib.rs`) set by a POSIX `SIGINT`/`SIGTERM` handler (installed without `SA_RESTART`). Both signals set the same flag; `INTERRUPT_SIGNAL` records which signal fired for correct exit codes (128+signum). Import loops, boot-wait loops, build operations, and all subprocess waits check it for clean Ctrl+C/SIGTERM cancellation. Second delivery of the same signal force-kills the process (handler restores `SIG_DFL`). Cleanup paths (e.g. stopping a container after boot failure in `sdme new`) use `save_and_reset_interrupt()` / `restore_interrupt()` to temporarily clear the flag for the cleanup operation and then restore it, so callers higher up the stack (e.g. `for_each_container`) still see the interruption and break out of loops. Multi-container loops (`for_each_container`, pod rm, fs rm) check `INTERRUPTED` after each action and break immediately; `check_interrupted()?` before the final error summary ensures the process exits with code 128+signum (e.g. 130 for SIGINT) and prints "interrupted, exiting". **When adding new batch/loop operations**, follow the same pattern: (1) `check_interrupted()?` at loop top, (2) `INTERRUPTED` check + break after each action, (3) `check_interrupted()?` before the `if failed` summary. The `auto_fs_gc` config key (default: `true`) controls whether operations clean up stale transactions from prior interruptions before proceeding; when `false`, only `sdme fs gc` cleans up.
- **Transactional staging**: mutating filesystem operations use enumerated staging directories named `.{name}.{kind}-txn-{pid}` (e.g. `.ubuntu.import-txn-42195`). On success, the staging dir is atomically renamed to its final location. On interruption or error, the staging dir is left behind; no cleanup runs during signal handling. The `Txn` type (`src/txn.rs`) encodes the operation kind and creator PID. `cleanup_stale_txns()` detects dead PIDs via `/proc/{pid}` and removes their artifacts; called automatically by `Txn::prepare()` when `auto_fs_gc` is enabled, or manually via `sdme fs gc`.
- **Rootfs export**: `sdme fs export <name>` defaults to container export; prefix with `fs:` for rootfs catalogue export (e.g. `sdme fs export fs:ubuntu out.tar.gz`). Smart error messages suggest the `fs:` prefix or its removal when the wrong kind is specified. Output formats: directory copy, tarball (uncompressed, gzip, bzip2, xz, zstd), or raw disk image (ext4 default, btrfs via `--filesystem btrfs`; bare filesystem for non-VM exports, GPT-partitioned for `--vm` exports). Format auto-detected from extension or overridden with `--fmt`. Container export: if running, reads directly from `merged/` with a consistency warning; if stopped, temporarily mounts a read-only overlayfs view (`mount_overlay_ro`) with an RAII `OverlayGuard` for safe unmount on drop. The unified `export()` function holds shared flock locks on the container and/or rootfs to prevent `sdme rm` / `sdme fs rm` from deleting resources during export. Stopped container exports use a `Txn` with `TxnKind::Export` as a marker for `fs gc` to detect stale overlay mounts from interrupted exports. Tar exports clean up partial output files on error. Raw image: sparse file formatted with `mkfs.ext4` or `mkfs.btrfs`, loop-mounted, tree copied, unmounted. Size auto-calculated as `max(256M, content * 1.5 + free_space)` or overridden with `--size` (which ignores `--free-space`). The `default_export_fs` config key sets the default filesystem type when `--filesystem` is not specified (default: `ext4`). The `default_export_free_space` config key sets the default extra free space for auto-calculated image size (default: `256M`). `--timezone` (e.g. `America/New_York`) sets the timezone in the exported rootfs by symlinking `/etc/localtime` to the zoneinfo file and writing `/etc/timezone`; validated against the rootfs zoneinfo database before export begins. Works with all output formats. Not needed for `fs import` since `systemd-nspawn` handles timezone at container boot via `--timezone=copy`.
- **VM export**: `sdme fs export --vm` prepares a GPT-partitioned raw disk image for booting as a standalone VM (e.g. cloud-hypervisor, QEMU). A Linux root partition is created via `sfdisk` at 1 MiB offset (standard GPT alignment), avoiding sector 0 conflicts with hypervisors. The filesystem is formatted on the partition device (`/dev/loopNp1`), not the whole image. Optional `--swap <size>` adds a second GPT partition (type=swap) formatted with `mkswap`; the swap entry is written to `/etc/fstab` as `/dev/vda2`. Modifications: serial console login via patched `serial-getty@.service` (strips `BindsTo=dev-*.device` that requires udev), `/etc/fstab` with `/dev/vda1` root device (and `/dev/vda2` swap if `--swap`), DHCP networking via `systemd-networkd`, hostname. Optional: `--dns` (without value copies host's `/etc/resolv.conf`; with IPs writes those as nameservers; omitted entirely leaves resolv.conf untouched), `--root-password` (SHA-512 crypt in `/etc/shadow`), `--ssh-key` (root authorized key), `--install-packages` (installs udev via chroot using `ChrootGuard` from `src/import/mod.rs`). Non-VM raw exports remain bare filesystem images (no partition table). NixOS VM export is not supported; use NixOS-native tooling (`nixos-generators`, `nixos-rebuild build-vm`) to build NixOS VM images.
- **Build COPY restrictions**: `sdme fs build` COPY writes through the merged overlayfs mount while the container stays running (hot copy), ensuring immediate visibility inside the container and correct overlayfs dcache behavior. `/tmp` is allowed in builds because the build container bind-mounts `upper/tmp` over nspawn's default tmpfs (nspawn always mounts tmpfs on `/tmp` before systemd starts, so masking `tmp.mount` alone is insufficient). The container is eagerly started before any ops so that `systemd-tmpfiles` cleanup completes before files are written to `/tmp`. `/run` and `/dev/shm` are still rejected. Opaque dirs are also rejected. COPY supports source prefixes: `fs:name:/path` reads from an imported rootfs, `container-name:/path` reads from another container (read-only overlay mount for stopped containers, merged view for running). `FROM` accepts an optional `fs:` prefix (`FROM fs:ubuntu`). Validation in `check_shadowed_dest()` (`src/build.rs`); errors include config file path and line number. **Resumable builds**: if a build fails at a RUN step, the build container's overlayfs upper layer is preserved. On re-run, the config file is hashed (SHA-256); if it matches the stored `BUILD_CONFIG_HASH`, completed steps (tracked by `BUILD_LAST_COMPLETED_OP`) are skipped and execution resumes. Config changes or `--no-cache` discard the stale container and start fresh. COPY source file changes are not tracked. Resource locking: builds hold shared `flock` locks on FROM rootfs and COPY sources; `sdme fs rm` and `sdme rm` acquire exclusive locks, preventing deletion of resources used by active builds.
- **Kubernetes Pod YAML**: `sdme kube` accepts `kind: Pod` (v1) and `kind: Deployment` (apps/v1; extracts pod template). Multi-container pods run as a single nspawn container with one systemd service (`sdme-oci-{name}.service`) per app container; all containers share the network namespace and can communicate via localhost. Rootfs is named `kube-{podname}` and built atomically via a staging directory. K8s command/args semantics: `command` overrides Docker ENTRYPOINT, `args` overrides Docker CMD. Volumes: emptyDir (directories at `/oci/volumes/{name}` inside the rootfs, bind-mounted into each app's root by a generated `sdme-kube-volumes.service` oneshot unit), hostPath (nspawn `--bind=` mounts to `/oci/volumes/{name}`), secret (files populated from `{datadir}/secrets/{name}/data/` into `/oci/volumes/{vol}/`), configMap (files populated from `{datadir}/configmaps/{name}/data/` into `/oci/volumes/{vol}/`), and persistentVolumeClaim (host dir at `{datadir}/volumes/{claimName}` bind-mounted to `/oci/volumes/{vol}/`). Secret and configMap volumes support `items` for projected key paths and `defaultMode` for file permissions. The volume mount service runs `mount --bind` in the container's PID 1 mount namespace before app services start (app units have `After=`/`Requires=sdme-kube-volumes.service`); read-only mounts get an additional `remount,ro,bind`. Env vars support `valueFrom` with `secretKeyRef` and `configMapKeyRef` for deferred resolution from secrets/configmaps at create time. `envFrom` bulk-imports all keys from a configMap or secret as env vars, with an optional `prefix`; explicit `env[]` entries take priority over `envFrom` for the same key. Ports are aggregated across all containers; `--private-network` is enabled when any ports are declared. Restart policy mapping: Always=always, OnFailure=on-failure, Never=no. State keys: `KUBE=yes`, `KUBE_CONTAINERS={csv}`, `KUBE_YAML_HASH={sha256}`. `kube delete` removes both the container and its kube rootfs. `sdme ps` shows a `kube:{container_names}` column for kube containers. For multi-container kube pods, `--oci APP` on `exec`, `join`, and `logs` requires the app name to select which container to target; single-container pods auto-select the only app when `--oci` is used without a value. Security operates at two layers: CLI flags (`--strict`, `--hardened`, `--drop-capability`, `--capability`, `--no-new-privileges`, `--read-only`, `--system-call-filter`, `--apparmor-profile`, `--userns`) apply at the **nspawn container level** (outer sandbox), while Pod YAML `securityContext` applies at the **OCI app service level** (inner systemd units). These are complementary and both can be used together.
- **Kubernetes probes**: Four probe types supported via an embedded `sdme-kube-probe` binary deployed at `/oci/.sdme-kube-probe` inside the container rootfs. All three probe types (startup, liveness, readiness) use systemd timer + oneshot service pairs that invoke the probe binary. No shell scripts or external tool dependencies (wget, bash). Startup probe: timer writes `/run/sdme-probe-startup-{name}.done` on success (after `successThreshold` consecutive successes); restarts service on threshold failure. Liveness probe: timer + service that restarts the OCI app service on failure threshold. Readiness probe: timer + service that writes `ready` or `not-ready` to `/oci/apps/{name}/probe-ready` (in overlayfs, readable from host for `sdme ps`); requires `successThreshold` consecutive successes to transition to ready. Liveness/readiness timers use `ConditionPathExists=/run/sdme-probe-startup-{name}.done` to gate on startup probe completion when present. Four probe mechanisms: `exec` (chroot + Command, std only), `httpGet` (raw HTTP/1.0 GET via TcpStream, supports custom `httpHeaders`, std only), `tcpSocket` (TcpStream::connect_timeout, std only), `grpc` (gRPC Health Checking Protocol via tonic). Timer units use `BindsTo=sdme-oci-{name}.service` to stop when the main service stops. Probe parameters: `initialDelaySeconds`, `periodSeconds`, `timeoutSeconds`, `failureThreshold`, `successThreshold`. The probe binary tracks both failure and success counters via files in `/run/`: failures reset the success counter and vice versa, matching Kubernetes semantics. Probe binary is automatically built by `build.rs` and embedded into sdme via `include_bytes!()`. `HAS_PROBES=yes` stored in container state. `sdme ps` health column shows `ready`/`not-ready` for kube containers with readiness probes. Probe check validation in `src/kube/plan.rs:build_probe_check()` returns `ProbeCheck` enum (`Exec`, `Http`, `Tcp`, `Grpc`). Unit generation in `src/oci/app.rs:generate_probe_units()`. Probe binary sources in `src/kube/probe/`.
- **Boot failure cleanup**: `sdme new` and `sdme kube apply` stop (but do not remove) the container on boot failure or Ctrl+C, leaving state on disk for debugging or `sdme fs gc`. `sdme start` stops the container on boot failure or Ctrl+C (preserving it on disk). All paths use `save_and_reset_interrupt()` before the stop operation so that `check_interrupted()` in the stop path does not short-circuit, then `restore_interrupt()` afterward so the interrupt propagates to callers.
- **Input sanitization**: sdme runs as root and handles untrusted input; hardening measures:
  - OCI tar paths: `..` rejected, leading `/` stripped (`sanitize_dest_path()` in `src/copy.rs`).
  - OCI digests: validated for safe characters before constructing blob paths (`resolve_blob()` in `src/oci/layout.rs`).
  - Rootfs names (`-r`): validated with `validate_name()` to prevent path traversal (`resolve_rootfs()` in `containers.rs`).
  - Opaque dir paths: must be absolute, no `..`, no duplicates; normalized before storage.
  - URL downloads: capped by `max_download_size` config key (default 50 GiB, `0` = unlimited).
  - Config files: written with explicit permissions (`0o600`).
- **Resource locking**: all operations that read or mutate containers, rootfs, pods, secrets, or configmaps use advisory `flock(2)` locks via `src/lock.rs` to prevent data races. **Shared locks** (read) allow concurrent operations (build, export, cp, start, create) and coexist with each other. **Exclusive locks** (write) protect destructive mutations (rm, fs rm, import, kube delete, pod rm) and block all other lock holders. All locks are **non-blocking** (`LOCK_NB`): if a lock cannot be acquired immediately, the operation fails with an error identifying the holder PID (read from the lock file at `{datadir}/locks/{kind}/{name}.lock`). Lock ordering to prevent deadlocks: `fs → containers → pods → secrets → configmaps`. Within the same kind, acquire shared before exclusive on different names. Locks are released automatically when the `ResourceLock` value is dropped (file descriptor closed); on process crash or SIGKILL the kernel releases the lock. **Note**: `stop` operates via D-Bus (`KillMachine`/`TerminateMachine`) and does not use flock, so stopping a container is never blocked by any lock. **When adding new operations**, follow the pattern: shared lock for reading/non-destructive access, exclusive lock for deletion/replacement.
- **File copy (`sdme cp`)**: copies files between host and containers/rootfs. One side must always be a host path. Container/rootfs paths must be absolute (after the colon). Recursive by default. Shares the same copy engine, path validation (`sanitize_dest_path`), and directory/file semantics as `fs build` COPY — when modifying one, keep both in sync. Running containers: read/write through `/proc/<leader>/root/` for full mount namespace access (sees tmpfs at `/tmp`, `/run`, `/dev/shm`). Userns containers (`--userns`/`--hardened`/`--strict`) fall back to `merged/` because the kernel blocks `/proc/<leader>/root/` traversal across user namespace boundaries; shadowed paths are rejected with an error suggesting `sdme exec`. Stopped containers: source reads via temporary read-only overlay mount; destination writes to `upper/` directly. Rootfs: direct read/write; running containers using the rootfs will not see changes until restarted (overlayfs lower-layer caching), and a warning is printed. Safety: device nodes refused when copying to host (use `--force`); setuid/setgid warned; shadowed dirs (`/tmp`, `/run`, `/dev/shm`) rejected for stopped container destinations (differs from build COPY which allows `/tmp` via a bind-mount trick). Shared flock locks prevent concurrent deletion during copy; see Resource locking design decision.
- **Filesystem fidelity**: `copy_tree` preserves hard links via `(st_dev, st_ino)` tracking in a `HardLinkMap`; hard-linked source files become hard links in the destination. Single-file `copy_entry` calls have no cross-file context and do not track hard links. Tar export preserves hard links as `EntryType::Link` entries and writes xattrs as `SCHILY.xattr.*` PAX extended headers via `builder.append_pax_extensions()`. The `read_xattrs()` function in `src/copy.rs` is shared between the filesystem copy path (`copy_xattrs`) and the tar export path. Sparse file preservation relies on the kernel's `copy_file_range(2)` syscall (used by `std::fs::copy` on Linux); chattr inode flags (`FS_IOC_GETFLAGS`) are not preserved as they are uncommon in container rootfs.
