## Resource Management

#### [RM-V1.30-1]: Background HNSW rebuild thread is detached — daemon shutdown cannot wait for it
- **Difficulty:** medium
- **Location:** `src/cli/watch.rs:965-1042` (`spawn_hnsw_rebuild`)
- **Description:** PR #1113 spawns a background HNSW rebuild via `std::thread::Builder::new().name(...).spawn(...)` and stores only the mpsc `Receiver` in `PendingRebuild` (line 1037-1041). The `JoinHandle` is dropped immediately, so the thread is detached. On `systemctl stop cqs-watch` (SIGTERM) the daemon's main loop exits and `WatchState` drops, but the rebuild thread (which holds its own `Store::open_readonly_pooled` handle plus a CUDA build pipeline at `commands::build_hnsw_index_owned`) keeps running until it sends on the channel — even though the receiver is already gone. On the user's A6000 a full 17k-chunk CUDA HNSW build is ~10-15s, plus another ~10-15s for the base index. Worst case: a `systemctl restart cqs-watch` triggers a fresh rebuild on the new daemon while the old daemon's orphaned rebuild thread is still spending GPU memory + CUDA streams on the now-discarded result. Two CUDA HNSW builds running concurrently against the same `index.db` snapshot is exactly the contention pattern that leads to `cuda_runtime` API errors (cuvs is process-global, not per-context). In addition, `index.hnsw.lock` may be re-acquired by the new daemon while the old thread still has it open via `Owned::save` — leading to a half-written graph file.
- **Suggested fix:** Hold the `JoinHandle` inside `PendingRebuild` next to `rx`. On daemon shutdown (the `loop` exit in `cmd_watch`), call `pending_rebuild.take().map(|p| p.handle.join())` with a bounded timeout (e.g. 2s) before letting the daemon exit. Better: have `spawn_hnsw_rebuild` accept a `Arc<AtomicBool>` cancellation flag, check it inside `build_hnsw_index_owned` between batches, and bail with `RebuildOutcome::Err(anyhow!("cancelled"))` so the GPU work stops promptly.

#### [RM-V1.30-2]: `pending_rebuild.delta` grows unbounded during a long rebuild window
- **Difficulty:** easy
- **Location:** `src/cli/watch.rs:611,623-626,2667-2674,2740-2741`
- **Description:** While a background HNSW rebuild is in flight (10-30s), every reindex cycle appends every newly-embedded chunk into `pending.delta: Vec<(String, Embedding)>` (line 2674). Each `Embedding` is `dim × 4 bytes` ≈ 4 KB on BGE-large (1024-dim). A bulk file operation during the rebuild — e.g. `git checkout` of a feature branch touching 5k files, `find -name '*.rs' -exec sed -i ...`, or a `cargo fix` pass — can push tens of thousands of chunks into the delta before the rebuild completes. 30k entries × 4 KB = 120 MB held in `WatchState`, on top of the rebuild thread's own working memory and the in-memory `state.hnsw_index`. There is no cap, no warn-on-overflow, no spill-to-disk fallback, and no early-cancel of the in-flight rebuild even when delta has already invalidated the snapshot it's building from.
- **Suggested fix:** Cap `delta` at e.g. `MAX_PENDING_REBUILD_DELTA = 5_000` entries (~20 MB on 1024-dim). On overflow: `tracing::warn!`, drop oldest entries (or all of them — the next `threshold_rebuild` will pick the chunks up from SQLite anyway), and mark the rebuild as `state.pending_rebuild = None` to short-circuit the swap entirely (the new in-memory index would be ~useless given the volume of misses). Add a structured event so operators can see when the daemon entered "delta saturation" mode.

#### [RM-V1.30-3]: `Embedder::new` opens a fresh `QueryCache` SQLite + 7-day prune on every CLI subcommand
- **Difficulty:** easy
- **Location:** `src/embedder/mod.rs:355-366`
- **Description:** `Embedder::new` unconditionally calls `QueryCache::open(&QueryCache::default_path())` followed by `c.prune_older_than(7)` (line 358-361), even on commands that will never call `embed_query` (e.g. `cqs notes list`, `cqs read`, `cqs slot list`, `cqs cache stats`, `cqs explain --no-embed`). The `QueryCache` is constructed lazily *during the embedder*, but the embedder itself is constructed eagerly by every CLI handler that takes a `cli.try_model_config()` (16 call sites in `Bash` cross-check above). Each open is: `std::fs::create_dir_all` + `chmod 0o700` + new tokio `current_thread` runtime + SqlitePool with `max_connections=1` + 4 PRAGMA + WAL setup + `prune_older_than(7)` (a `DELETE` against `query_cache`). Cold start of `cqs --help` (which already lazy-loads the embedder for completion) pays this; a hot `cqs notes list` run via daemon doesn't, but a CLI bypass (`CQS_NO_DAEMON=1`) does. On WSL DrvFS this is ~30-50ms of completely wasted I/O per invocation.
- **Suggested fix:** Lazy-open the disk cache the first time it's actually used inside `embed_query`. Store it as `OnceLock<Option<QueryCache>>` rather than constructing eagerly in `new_with_provider`. The in-memory `LruCache` stays as-is; only the SQLite-backed half pays the cold open. Bonus: cli subcommands that are pure SQL (notes, slot, cache, telemetry) no longer touch `~/.cache/cqs/query_cache.db` at all.

#### [RM-V1.30-4]: `LocalProvider::stash` retains all submitted batch results until provider drop
- **Difficulty:** medium
- **Location:** `src/llm/local.rs:74,304-309,542-547`
- **Description:** `LocalProvider::submit_via_chat_completions` (PR #1101) stores each submitted batch's results in `self.stash: Mutex<HashMap<String, HashMap<String, String>>>`, keyed by batch_id. The only path that removes entries is `fetch_batch_results` (line 545: `stash.remove(batch_id)`). If a caller submits multiple batches and crashes (or panics) between submit and fetch, all unfetched batches stay resident for the lifetime of the `LocalProvider`. Even on the happy path, a long-running `cqs index --llm-summaries` over a 5k-chunk corpus with `concurrency=4` may submit several batches sequentially: each batch ~250 successful items × 500-byte summary text ≈ 125 KB stashed at a time, but if a batch fails the entries-so-far are retained anyway because the function returns `Err(LlmError::Api{...})` after inserting partial results into the stash (line 304-309 happens unconditionally before the auth-fail bail at 286). Worst case for a 50k-chunk doc-comments pass with 50% timeout failures: ~25k summaries × ~500 bytes = ~12 MB of dead text held in the provider until the CLI exits. There is no LRU, no TTL, no "drain everything older than N" sweep.
- **Suggested fix:** (1) Move the stash insert past the auth-fail bail so failed batches don't leak partial results. (2) Add `LocalProvider::drain_old_batches(&self, max_keep: usize)` called from the outer `llm_summary_pass` after each `wait_for_batch + fetch_batch_results` cycle. (3) On the auth-fail Err arm at line 286, explicitly clear the stash entry. (4) Cap total stash size: if it exceeds e.g. 1000 batches or 100 MB cumulative, evict oldest by insertion order.

#### [RM-V1.30-5]: Daemon never checks `fs.inotify.max_user_watches` — silently drops events on large monorepos
- **Difficulty:** medium
- **Location:** `src/cli/watch.rs:1947-1949`
- **Description:** `RecommendedWatcher::new(...).watch(&root, RecursiveMode::Recursive)` walks the tree and registers an inotify watch per directory. Linux's `fs.inotify.max_user_watches` defaults to 8192 on older distros, 65536-1048576 on newer ones. On a project with deep `node_modules/`, `target/`, `vendor/`, or `dist/` directories — none of which are filtered before `watch()` because `notify` registers everything before our gitignore filter sees it — the limit can be silently exceeded. `notify` returns `Err` on registration failure for individual sub-directories but the `watch(root, Recursive)` call at line 1949 only returns the *root* registration error. Per-subdir failures are swallowed, leading to "watch isn't picking up changes in src/foo/" with zero diagnostic output. The `--poll` fallback exists but the daemon never auto-detects exhaustion to suggest it.
- **Suggested fix:** At daemon startup, read `/proc/sys/fs/inotify/max_user_watches` and `/proc/<pid>/status` (Inotify count). If the project's directory count is within 90% of the limit, log a `tracing::warn!` recommending `--poll` or a `sysctl fs.inotify.max_user_watches=524288` bump. For inotify-watcher, replace the direct `watch(&root, Recursive)` with a manual descent that respects the gitignore matcher: only watch directories that wouldn't be ignored anyway (avoids registering target/, node_modules/, .git/). This both reduces inotify pressure and matches the user's expectations about which paths the daemon should react to.

#### [RM-V1.30-6]: `select_provider()` triggers CUDA/TensorRT probe + symlink ops for every CLI process, including no-embed commands
- **Difficulty:** medium
- **Location:** `src/embedder/provider.rs:171-248`, `src/embedder/mod.rs:312-313`
- **Description:** PR #1120 (Phase A) refactored execution-provider detection but kept the call-site contract: `Embedder::new` → `select_provider()` → `detect_provider()` → `ensure_ort_provider_libs()`. Even with `CACHED_PROVIDER: OnceCell` memoizing the result, the *first* call within a process always: walks `~/.cache/ort.pyke.io/dfbin/<triplet>/`, sorts directory entries, reads `/proc/self/cmdline`, joins multiple `PathBuf`s, calls `dunce::canonicalize` on each library, and `std::os::unix::fs::symlink`s up to 3 `.so` files into both `ort_search_dir` and the first writable `LD_LIBRARY_PATH` entry. On a CLI invocation that never reaches `embed_query` (e.g. `cqs notes list`, which constructs an embedder via `try_model_config`), this is pure waste — the CUDA probe `ort::ep::CUDA::default().is_available()` itself can take 200-500ms because it lazily loads `libcudart.so` and pings the driver. Phase A's `ep-coreml` / `ep-rocm` cfg-gates are correct in spirit but they're cfg-gates over branches that today are dead `tracing::warn!` arms — the `ensure_ort_provider_libs` always fires regardless.
- **Suggested fix:** Move the provider probe + symlink work into a `LazyLock<ExecutionProvider>` that fires only inside `Session::create_session(model_path, ...)` — i.e. on the first actual ONNX inference, not on every `Embedder::new`. Construction-time should know the *target* provider (from CLI flag / env) but defer side effects until needed. The `Mutex<Option<Session>>` already supports lazy session init; making provider detection equally lazy keeps no-embed commands free of the GPU probe + symlink overhead.

#### [RM-V1.30-7]: `LocalProvider::submit_via_chat_completions` worker threads use default 2 MB stack — `concurrency=64` allocates 128 MB just for the fan-out
- **Difficulty:** easy
- **Location:** `src/llm/local.rs:163-256` (`std::thread::scope` in `submit_via_chat_completions`)
- **Description:** `local_concurrency()` clamps `CQS_LOCAL_LLM_CONCURRENCY` at `[1, 64]`. At the upper end, `std::thread::scope` spawns 64 worker threads via `s.spawn(...)` (line 176) with the platform default stack (2 MB on glibc) — 128 MB of stack space just for batch fan-out, on top of the per-thread `reqwest` blocking client state and JSON parsing buffers. The worker body is shallow (channel recv → reqwest send → JSON parse → mutex insert → callback) so 256 KB or 512 KB would suffice. The cap is also surprisingly high: 64 concurrent HTTPS connections to a local llama.cpp server is going to thrash the GPU, not parallelize — the practical sweet spot for `vllm` on an RTX 4000 is 4-8.
- **Suggested fix:** Use `std::thread::Builder::new().stack_size(512 * 1024).name(format!("cqs-llm-worker-{i}"))` inside a manual scope (or `scope.builder()` if one stabilizes — currently scoped threads use `Scope::spawn` without a stack-size hook, so swap to a `std::thread::Builder` + manual join Vec). Drop the upper clamp from 64 to 16; at >16 the local server is the bottleneck anyway.

#### [RM-V1.30-8]: `serve` handlers spawn one `tokio::task::spawn_blocking` per HTTP request, default blocking pool is 512 threads (~1 GB stack)
- **Difficulty:** medium
- **Location:** `src/serve/handlers.rs:86-89,100+,...` (every handler has a `spawn_blocking`), `src/serve/mod.rs:92-95`
- **Description:** Every `cqs serve` route handler wraps its `Store` call in `tokio::task::spawn_blocking(...)` (six occurrences; at lines 86, ~100, ~140, ~180, ~210). The runtime is built via `Builder::new_multi_thread().enable_all().build()` (mod.rs:92-95) which uses tokio's defaults: `worker_threads = num_cpus` (already noted in RM-V1.29-6) and `max_blocking_threads = 512`. A hostile or buggy client opening 512 parallel `/api/graph` requests (loading a 50k-node graph response each) can saturate the blocking pool — each blocking thread holds the default 2 MB Linux stack + the SQL working set (`HashMap::with_capacity(rows.len())` for up to 50k rows = several MB). Worst case: 512 × (2 MB stack + ~10 MB working set) ≈ 6 GB. The fact that v1.30.0 added per-launch auth (#1118) closes the trivial unauthenticated DoS vector, but an authenticated user (or a malicious browser tab on the same host once they've grabbed the cookie) can still launch this.
- **Suggested fix:** `.max_blocking_threads(8)` on the `Builder` — 8 concurrent SQL queries is more than enough for an interactive single-user UI. Combined with `.worker_threads(4)` from RM-V1.29-6 the daemon's max steady-state thread count is bounded at 12, vs. 512+num_cpus today. Optionally also wrap the inner `Store::rt.block_on` calls with a `tokio::time::timeout(30s, ...)` so a stuck SQL query doesn't pin a blocking thread indefinitely.

#### [RM-V1.30-9]: `LocalProvider::http` uses default reqwest connection pool (no idle cap, no per-host limit)
- **Difficulty:** easy
- **Location:** `src/llm/local.rs:97-100`
- **Description:** `Client::builder().timeout(timeout).redirect(Policy::none()).build()` — no `pool_max_idle_per_host`, no `pool_idle_timeout`, no `tcp_keepalive`. reqwest's blocking client defaults to `pool_max_idle_per_host = usize::MAX` and `pool_idle_timeout = 90s`. On a long-running `cqs index --llm-summaries` that submits batches over hours, the connection pool to the local server can grow without bound — particularly bad against vLLM, which is single-process and sees each idle keep-alive as a held slot in its connection table. If the daemon embeds `LocalProvider` (it doesn't today, but the LLM path is converging on daemon execution), pool growth becomes permanent.
- **Suggested fix:** `Client::builder().pool_max_idle_per_host(self.concurrency).pool_idle_timeout(Duration::from_secs(30)).timeout(timeout)...`. Caps the idle pool at the worker count (extra idle connections beyond `concurrency` are by definition unused) and recycles after 30s of idle.

#### [RM-V1.30-10]: `Mutex<Option<Arc<Tokenizer>>>` pattern in `Embedder` keeps a stale tokenizer Arc alive on `clear_session` if any inference is in flight
- **Difficulty:** medium
- **Location:** `src/embedder/mod.rs:261,808-823`
- **Description:** `clear_session` sets `*self.tokenizer.lock() = None` (line 820-821), but the comment on line 256-258 explicitly says *"Arc holds a strong ref so in-flight inference that grabbed an Arc clone before this call continues with its own copy."* This is correct for safety, but the implication is that if a long-running inference (e.g. a batch of 1000 chunks at 4ms each = 4s of work) is mid-flight when `sweep_idle_sessions` fires, the tokenizer Arc stays in memory until that batch completes — the BGE-large tokenizer is ~10 MB, larger BPE vocabularies are ~20 MB. Concurrently, the next inference call after `clear_session` lazy-reloads the tokenizer (10-30 ms), so for a brief window the daemon holds *two* tokenizer copies. Combined with the parallel session reload (~500 MB), the actual peak memory during the "clear → next-use" handoff is ~1 GB rather than the documented ~500 MB. This is documented behavior, but the actual lifecycle isn't surfaced anywhere — the idle-eviction path looks like a clean drop-and-reload to the operator.
- **Suggested fix:** Either (a) wait for in-flight inference before clearing — adds a `RwLock` around the tokenizer Arc and `clear_session` takes the write lock, blocking until inference releases its read lock; or (b) document the doubled-memory window in the `clear_session` doc comment and surface it via a `tracing::info!(stage = "clear_during_inference", ...)` event when `Arc::strong_count(&tok) > 1` at clear time so operators can correlate memory spikes. Option (b) is the lower-risk fix since (a) extends the inference critical section.

## Summary
Found 10 resource-management issues new in v1.30.0:
- (1) PR #1113 detached HNSW rebuild thread can outlive daemon shutdown and contend on GPU/index lock with the next daemon's rebuild;
- (2) sibling issue: `pending_rebuild.delta` is unbounded — bulk git operations during a rebuild window can pin 100MB+;
- (3) `Embedder::new` opens a fresh `QueryCache` SQLite + 7-day prune on every CLI subcommand even when no embedding ever happens;
- (4) PR #1101's `LocalProvider::stash` retains all submitted batch results until provider drop, with no LRU/TTL/cap;
- (5) inotify watcher silently drops events on large monorepos — daemon never checks `fs.inotify.max_user_watches` or warns;
- (6) PR #1120's `select_provider()` still fires CUDA probe + symlink work eagerly via `Embedder::new` even on no-embed commands;
- (7) `LocalProvider` worker threads use default 2MB stack, allocating 128MB at `concurrency=64`;
- (8) `cqs serve` handlers `spawn_blocking` without `max_blocking_threads` cap — authenticated user can pin 512 threads × ~10MB working set;
- (9) `LocalProvider::http` reqwest client has no pool_max_idle / idle_timeout, can leak idle connections to local server over a long indexing run;
- (10) `Embedder::clear_session` documents but doesn't surface the doubled-memory window when in-flight inference holds a tokenizer Arc concurrent with a session reload.
