# Inkhaven 1.2.15 — Stability + Self-Healing

Status: **proposal**, written 2026-05-31.
Cycle target: 4-6 weeks.
Tests target: `637 → ≥ 700` (+63 new regression
tests pinning surfaced behaviour at panic sites,
plus integration tests for health + doctor + crash
report).

## 0. Why a stability release

The 1.2.4 → 1.2.14 cycles added significant
surface area: nine system books, six AI inference
flows, three system-book CLIs, fifteen modal
overlays, an embedded shell + scripting VM, two
standalone editor TUIs (config + prompts), Typst
in-process compilation, a thread-weave swim-lane
renderer, sidecar-JSON comments, four export
formats.  Every feature carries its own
`.unwrap()` cluster, its own lock contention
points, its own assumed-coherent state.

1.2.15 pays that down.  Goal: **zero panics from
the hot path**, **every unsaved buffer
recoverable across crashes**, **every project-
wide inconsistency either auto-fixed or surfaced
to the user**.

The "0% crashes / 0% data loss" goal is
asymptotic — it converges, never arrives.  This
cycle's success criteria are concrete:

* **No panic in the TUI event loop** under fuzz
  (random key sequences + random file states).
* **No data loss** when the process is `kill -9`d
  with unsaved edits, as long as the autosave
  cadence has fired at least once.
* **Crash report covers every panic that DOES
  fire**, with enough state to lift the user back
  to the moment-before.
* **Doctor scan catches every class of
  inconsistency** we've ever fixed manually.
* **Health monitor auto-repairs the safe classes**
  before the user notices.

## 1. Themes

### Theme S — Stability sweep

Hot-path hardening.  Catalogue every panic site
(`.unwrap()`, `.expect()`, `panic!()`,
`unreachable!()`, `unimplemented!()`, `vec[i]`,
`.unwrap_or_default()` that masks real errors,
poisoned-lock unwraps).  Triage hot vs. cold;
replace hot-path panics with `Result` plumbing or
explicit graceful degradation.

### Theme H — Background health monitor

A tokio task that runs periodic integrity checks
on the DB, the indices, and the in-memory state.
Status-bar `health` chip.  Cheap default cadence
(15 min), tighter on-idle pass, kill switch via
HJSON.  Safe auto-repair only — anything
destructive surfaces to the user as a doctor
finding instead.

### Theme D — Doctor scan + autofix

Extend `inkhaven doctor` from its current
keybind-cache + filter-words + voices role into
a project-wide problem scan.  Detect zero-byte
files, orphan paragraphs, missing referenced
files, corrupt sidecar HJSON, dangling wiki-
links, orphan threads/waypoints, mismatched DB-
vs-disk state.  `--autofix` flag with per-class
confirmation; TUI surface via `Ctrl+B Shift+?`.

### Theme R — Crash report writer + recovery

Custom panic hook.  When anything panics, flush
pending writes, serialise the project state +
panic context + recent action ring to
`inkhaven-crash-<ts>.hjson` in cwd, then exit.
New `inkhaven recover <crash.hjson>` CLI walks
the report and offers to restore unsaved
buffers.

## 2. Phase S — Stability sweep

### 2.1 Survey methodology

```bash
# Catalogue every panic site
rg --type rust -n '\.unwrap\(\)|\.expect\(|panic!\(|unreachable!\(|unimplemented!\(' src/ \
  > target/panic-sites.txt

# Catalogue index-into-slice sites (potential out-of-bounds)
rg --type rust -n '\[[a-z_]+\]\b' src/ \
  | rg -v ' (str|\&str|String)' \
  > target/index-sites.txt

# Catalogue lock-acquire sites (poisoned-lock unwrap)
rg --type rust -n '\.lock\(\)|\.read\(\)|\.write\(\)' src/ \
  > target/lock-sites.txt
```

Expected scale (best-guess from a 13K-LOC TUI
crate + cli/store sibling crates): ~600-1200
`.unwrap()` sites, ~150 `.expect()`, ~50
`unreachable!()` / `panic!()`, ~80 lock-acquire
sites.

### 2.2 Triage classes

Every site falls into one of five classes:

| Class | Action |
|-|-|
| **Hot path — user-facing** | Replace with `Result` or graceful degradation.  Examples: every TUI event-handler, every file save, every DB query in the autosave loop, every render call. |
| **Hot path — internal invariant** | Keep panic but document the invariant in a `// invariant:` comment + add a debug-only assert.  Example: `Vec::pop` after just-pushed. |
| **Cold path — initialisation** | Keep panic (init failure is genuinely fatal) but ensure the panic message names the underlying cause + the config file/key that triggered it. |
| **Test code** | Skip.  Test panics are fine. |
| **Examples / docs** | Skip. |

### 2.3 Hardening patterns

Five idioms to replace hot-path panics:

* **`Result<T, anyhow::Error>` + `?`** — when the
  caller can plausibly recover.
* **`Result<T, ErrorChip>` + status-bar surface**
  — when the user should see "what went wrong"
  but the TUI should keep running.
* **`Option<T>` + early-return** — when "missing"
  is a normal state, not an error.
* **Lock-poisoning recovery** — `Mutex::lock()`
  → `match` on poison; on poisoned, log + recover
  the inner value via `into_inner()`; never
  `.unwrap()` a lock guard.
* **Saturating arithmetic** — `i.saturating_sub(1)`
  / `i.checked_add(1)` for cursor / scroll
  arithmetic.  No more `usize` underflow panics
  when the editor is empty.

### 2.4 Test gates

* **Fuzz the TUI event loop.**  Property test:
  feed N random keystrokes + N random tree
  operations + N random AI events to a headless
  App; assert no panic.  N ranges 100 → 10000.
* **Fuzz the save pipeline.**  Property test:
  random text + random project state +
  randomly-interrupted save; assert no data loss.
* **Lock-poisoning regression tests.**  Force a
  poisoned mutex; assert the app recovers
  without panic.

Phase S commits as ~3-5 PRs (one per module
group: tui/app/*, tui/render/*, store/*, cli/*,
language/threads/comments-impl).  Each PR ships
the sweep + the regression tests that pin the
new behaviour.

## 3. Phase H — Background health monitor

### 3.1 Architecture

A single tokio task spawned from `main` after the
project opens.  Holds a `mpsc::Sender<HealthEvent>`
back to the TUI; the TUI consumes events on its
main select loop and updates the status-bar chip.

```rust
pub enum HealthEvent {
    Ok,                              // tick clean
    Warning(HealthFinding),          // log + chip
    Repaired(HealthFinding, String), // auto-fixed
    Error(HealthFinding),            // user must intervene
}

pub struct HealthFinding {
    pub class: HealthClass,    // Db / Index / Vectors / Editor / Tree / Disk
    pub severity: Severity,    // Info / Warn / Error / Critical
    pub detail: String,        // human-readable
    pub auto_repairable: bool,
}
```

### 3.2 Integrity checks

| Check | Cadence | Auto-repair? |
|-|-|-|
| DuckDB `PRAGMA integrity_check` | Every 15 min | No — surface as Critical |
| Tantivy index integrity | Every 15 min | Yes — rebuild from DB |
| HNSW vector count vs. DB row count | Every 15 min | Yes — re-embed missing |
| Open paragraph textarea vs. disk | Every 30 sec | Yes — pick the newer (with `autosave_resolution_policy` knob) |
| Tree parent-pointer integrity | Every 30 sec | Yes — re-link by slug |
| Disk space < 10 % free | Every 5 min | No — surface as Critical |
| Backup older than `backup.max_age` | Every 5 min | No — surface as Warn |

### 3.3 Auto-repair safety policy

* **Never auto-repair when the user is in the
  edit path of the affected resource.**  E.g., if
  the open paragraph's disk file just changed and
  the textarea is dirty, log + chip — don't
  silently overwrite.
* **Always log every repair to
  `.inkhaven/health.log`** with timestamp + before
  + after.  Logs rotate at 1 MB; keep the last 5.
* **Repairs are idempotent** — running the same
  check twice always yields the same outcome.

### 3.4 Status-bar chip

`[health: ✓]` (green) — last tick clean.
`[health: ✎]` (amber) — last tick repaired.
`[health: ⚠]` (yellow) — warning open.
`[health: ✗]` (red) — error open (click chip /
`Ctrl+B Shift+H` to open health detail panel).

Note: `Ctrl+B Shift+H` currently maps to the
sentence-rhythm gauge.  Rebind candidates:
`Ctrl+B Shift+J` (J for journal/health log) or
`Ctrl+B Shift+0` (0 for "system health" — pairs
with `Ctrl+B 0` config-editor mental model).

### 3.5 HJSON knobs

```hjson
{
  health: {
    enabled: true
    cadence_seconds: {
      integrity: 900     // 15 min
      sync: 30           // 30 sec
      disk: 300          // 5 min
    }
    auto_repair: {
      tantivy_index: true
      hnsw_vectors: true
      tree_parent_pointers: true
      disk_textarea_sync: false  // default off — user must opt in
    }
    log_rotation_mb: 1
  }
}
```

## 4. Phase D — Doctor scan + autofix

### 4.1 Problem catalogue

| Class | Detection | Autofix |
|-|-|-|
| **Zero-byte `.typ` file** | walk project tree; `metadata().len() == 0` | Delete the paragraph row from DB + the file from disk |
| **Orphan paragraph** | DB row with no on-disk file | Delete the DB row + remove from index |
| **Missing referenced file** | DB row pointing at `<path>` that doesn't exist | Same — delete row |
| **Corrupt sidecar HJSON** | `.comments.json` / `.bund` / language-template HJSON that fails to parse | Move to `<file>.corrupt-<ts>.bak`, log loudly |
| **Dangling wiki-link** | `[[slug]]` in prose where `slug` doesn't resolve to any paragraph | Surface only — never auto-edit user prose |
| **Orphan thread waypoint** | Waypoint in `Threads` book linking to nonexistent paragraph | Delete the waypoint paragraph |
| **Mismatched DB / disk hash** | DB has stored `content_hash` that doesn't match file's current hash | Re-hash + update DB |
| **Mismatched FTS index** | DB row exists, Tantivy missing it | Re-index that row |
| **Mismatched vector index** | DB row exists, HNSW missing it | Re-embed |

### 4.2 CLI surface

```bash
$ inkhaven doctor                    # current behaviour + new scan
$ inkhaven doctor --autofix          # repair safe classes
$ inkhaven doctor --autofix --yes    # don't prompt; useful for CI
$ inkhaven doctor --json             # machine-readable findings
$ inkhaven doctor --class <name>     # scan only one class
```

### 4.3 TUI integration

New chord `Ctrl+B Shift+?` (or `Ctrl+B Shift+0`,
TBD) opens a `Doctor` modal — sortable table of
findings with `r` to repair the highlighted
finding, `R` to repair all repairable, `Esc` to
close.  Each finding has a "why this matters"
line + a "repair plan" line.

### 4.4 Per-class confirmation flow

Repairs that touch user content (`.typ` file
delete, sidecar move) always confirm in the TUI.
Repairs that touch derived state (index, hash,
vectors) confirm only if the project is dirty
elsewhere.  `--autofix --yes` bypasses; the
findings + actions still log to
`.inkhaven/doctor.log`.

## 5. Phase R — Crash report writer + recovery

### 5.1 Panic-hook design

Install in `main` before the runtime starts:

```rust
std::panic::set_hook(Box::new(|info| {
    // 1. Flush pending writes — try to autosave
    //    every dirty buffer to <path>.inkhaven-rescue
    //    next to the original.
    let _ = crate::rescue::flush_dirty_buffers();

    // 2. Capture panic context.
    let report = crate::crash::CrashReport::capture(info);

    // 3. Serialise to HJSON.
    let path = std::env::current_dir()
        .unwrap_or_else(|_| PathBuf::from("."))
        .join(format!("inkhaven-crash-{}.hjson",
                      chrono::Utc::now().format("%Y%m%dT%H%M%S")));
    let _ = report.write_atomic(&path);

    // 4. Restore terminal state.
    let _ = crate::tui::restore_terminal();

    // 5. Print a one-line breadcrumb.
    eprintln!("inkhaven crashed — report written to {}", path.display());
}));
```

### 5.2 Crash-report HJSON schema

```hjson
{
  version: "1.2.15"
  generated_at: "2026-05-31T14:23:00Z"
  panic: {
    message: "called Option::unwrap() on None"
    location: "src/tui/app/render/modals.rs:412"
    thread: "main"
    backtrace: [ ... ]                  // captured if RUST_BACKTRACE=1
  }
  project: {
    path: "/Users/gandalf/Books/my-novel"
    open_book: "manuscript"
    open_paragraph: "01-chapter-1/01-opening"
  }
  buffers: [
    {
      paragraph: "01-chapter-1/01-opening"
      dirty: true
      cursor: { row: 12, col: 4 }
      rescue_path: "/Users/gandalf/Books/my-novel/manuscript/01-chapter-1/01-opening.typ.inkhaven-rescue"
      bytes: 4823
    }
  ]
  recent_actions: [
    { ts: "2026-05-31T14:22:58Z", action: "view.add_comment", details: { selection_start: 100, selection_end: 145 } }
    { ts: "2026-05-31T14:22:55Z", action: "ai.continuation_draft" }
    // ring buffer of last 50 actions
  ]
  environment: {
    os: "darwin 25.4.0"
    term: "xterm-256color"
    inkhaven_version: "1.2.15"
    dep_versions: { ratatui: "0.30", crossterm: "0.30", ... }
  }
  process: {
    pid: 12345
    uptime_seconds: 3247
    memory_rss_bytes: 184529920
  }
}
```

### 5.3 Atomic write flow

* Write to `<target>.tmp`.
* `fsync` the file handle.
* `rename` `<target>.tmp → <target>`.
* `fsync` the parent dir on Unix (Windows skips).

The rescue buffers (`*.inkhaven-rescue`) follow
the same flow — atomic temp + rename + fsync.

### 5.4 `inkhaven recover <crash.hjson>` CLI

```bash
$ inkhaven recover inkhaven-crash-20260531T142300.hjson
Found 3 rescue buffers in the report.

  [1] manuscript/01-chapter-1/01-opening.typ
      rescue: 4823 bytes (dirty at crash time)
      current on-disk: 4801 bytes
      mtime delta: +22 bytes since last save
      apply? [y/N/diff]

  [2] manuscript/02-chapter-2/03-rain.typ
      rescue: 891 bytes (dirty at crash time)
      current on-disk: 891 bytes (identical)
      no recovery needed.

  ...
```

`diff` shows a unified diff between rescue and
on-disk.  `y` applies the rescue (with an
automatic snapshot capturing the on-disk version
first, so the user can always roll back).  `N`
skips.  After the walk, the crash report file +
rescue buffers are moved to `.inkhaven/recovered/`
unless `--keep` is passed.

### 5.5 What gets captured / what doesn't

**Captured:**

* Panic message + location + (best-effort)
  backtrace
* Currently open project path + book + paragraph
* Every dirty editor buffer (rescued separately
  as `.inkhaven-rescue` files)
* Recent action ring (last 50 user actions, with
  args)
* Environment fingerprint (OS, term, version, dep
  versions)
* Process stats (pid, uptime, RSS)

**Deliberately NOT captured** (privacy + size):

* LLM prompts / responses
* Search queries
* Snapshot bodies
* The full DuckDB / Tantivy / HNSW state (too
  large; the recover CLI re-derives from rescue
  buffers + disk)

## 6. Phased rollout

| Phase | Scope | Tests |
|-|-|-|
| **R.1** | Panic hook + crash-report writer + atomic flush | ~6 |
| **R.2** | `inkhaven recover` CLI + rescue-buffer pickup | ~5 |
| **S.1** | Sweep `src/tui/app/**` (event loop + dispatch) | ~12 |
| **S.2** | Sweep `src/tui/app/render/**` (every render call) | ~8 |
| **S.3** | Sweep `src/store/**` + `src/cli/**` | ~10 |
| **S.4** | Sweep `src/tui/snippets.rs`, `src/tui/comments.rs`, `src/tui/threads/**`, `src/tui/language/**` | ~10 |
| **S.5** | Lock-poisoning recovery + saturating arithmetic | ~6 |
| **H.1** | Health-monitor scaffold + status-bar chip | ~4 |
| **H.2** | Integrity-check implementations (DB / index / vectors) | ~6 |
| **H.3** | Auto-repair flow + log rotation | ~4 |
| **D.1** | Doctor scan classes + `--json` output | ~8 |
| **D.2** | `--autofix` flow + per-class confirmation | ~5 |
| **D.3** | Doctor TUI modal + `Ctrl+B Shift+?` chord | ~3 |

Total target: **~87 new tests**, on top of the
637 from 1.2.14.  Each phase ships as its own
commit; main stays green between them.

Recommended order:

1. **R.1 + R.2** first — every subsequent panic
   becomes survivable.  Even before we reduce the
   panic surface, the user can recover.
2. **S.1 → S.5** in module groups — reduce panic
   surface incrementally.  Pair each sweep with
   the regression tests that pin the new
   behaviour.
3. **H.1 → H.3** — health monitor lands on top of
   the harder code.
4. **D.1 → D.3** — doctor lands last because it
   reuses H's classification machinery.

## 7. Risks + open questions

* **Auto-repair vs. user intent.**  A
  deliberately-empty paragraph is indistinguishable
  from a zero-byte bug from disk alone.  Mitigation:
  doctor surfaces the finding; autofix is opt-in;
  health monitor never auto-deletes user files.

* **Rescue buffers leak.**  If the user dismisses
  the recover CLI, do `.inkhaven-rescue` files
  pile up?  Mitigation: doctor scan flags rescue
  files older than 7 days; `inkhaven doctor
  --autofix` deletes them.

* **Panic-hook re-entry.**  If the panic hook
  itself panics (e.g., disk full while writing
  the report), the user loses the report.
  Mitigation: every step in the hook is wrapped
  in `let _ = ...` — failures are silent; the
  hook always falls through to the breadcrumb
  + `restore_terminal`.

* **Backtrace capture cost.**  Capturing a
  full backtrace on every panic in release mode
  is expensive (10-100 ms).  Mitigation: only
  capture when `RUST_BACKTRACE=1` is set OR when
  the panic is in a known-flaky surface.  In
  prod, the recent-action ring + the rescue
  buffers are usually enough.

* **Health-monitor false positives.**  The 30-sec
  textarea-vs-disk sync check will fire on every
  external `git checkout`.  Mitigation:
  `auto_repair.disk_textarea_sync` defaults off;
  the chip shows the divergence but doesn't
  auto-overwrite.

* **Doctor scan on huge projects.**  A 10K-file
  project takes time to walk.  Mitigation: scan
  is incremental (resume from last position);
  `--class <name>` runs only one class;
  background health monitor handles the high-
  cadence checks separately.

* **Lock-poisoning recovery semantics.**  When a
  lock is poisoned, the inner state may be
  corrupt.  Recovering "into_inner" is sometimes
  unsafe.  Mitigation: classify each lock by
  whether its state can be recovered; for
  unrecoverable ones, log + bail (which now means
  "write crash report and exit" — survivable).

## 8. Out of scope (deferred to 1.2.16)

* **Phase G — Git/backup safety rails.**
  Uncommitted-changes warning at exit, backup-
  freshness chip, disk-space gate.  Compounds
  with Phase H but adds another modal + status
  chip; defer to keep 1.2.15 scope crisp.

* **Distributed-write conflict handling.**  When
  the same project is open in two TUIs (e.g.,
  via SSH multiplex), writes can collide.
  Survivable today via snapshots; principled
  handling is a 1.2.16+ concern.

* **Differential rescue.**  Currently we write
  the whole buffer to the rescue file.  Future
  optimisation: write only the diff against the
  last-saved version.  Wait until we have data
  on how big rescue files actually are.

## 9. Recommendation

Ship in this order:

1. **R.1 + R.2** (crash report + recover CLI) —
   the safety net.  Every subsequent panic becomes
   survivable.
2. **S.1 → S.5** (stability sweep, module by
   module) — reduce the panic surface so R fires
   less.
3. **H.1 → H.3** (health monitor) — catch
   inconsistencies before they cause a panic in
   the first place.
4. **D.1 → D.3** (doctor scan) — surface anything
   H can't auto-fix; user-driven cleanup.

If the cycle runs long, the natural cut points
are: defer D.3 (TUI modal) to 1.2.16 if D.1 + D.2
(CLI) cover the use case; defer S.4 if the
specific modules involved (snippets / comments /
threads / language) prove cleaner than expected.
R.1 + R.2 + S.1 + S.3 + H.1 + H.2 alone is
already a coherent stability release.

The cycle's exit criteria:

* Zero panics in the property-test fuzz suite at
  N = 1000.
* No data loss in the kill-9-with-autosave-on
  regression test.
* Doctor catches every inconsistency class we've
  documented under "Carryovers" across the 1.2.x
  cycle.
* Health monitor's auto-repair rate ≥ 95 % of
  detected issues.
* Crash reports parse + recover cleanly on every
  panic surfaced during fuzz.

When all five hold, we cut.
