# `Session` refactor — planning document

This document plans the introduction of a `Session` (a.k.a. compilation /
workspace / driver) layer above the existing scanner/parser/interpreter stages.

The goal is to move project-level state — source files, import resolution,
search paths, loaded-file dedup, the global instance-id counter, accumulated
diagnostics — out of the `Interpreter` (where some of it lives today by
accident) and into one explicit owner.

The three pipeline stages themselves stay structurally as they are. What
changes is what sits _above_ them and what they have to carry around.

---

## 1. Motivation

The current pipeline shape is fine for the simple "scan → parse → interpret
one program, then exit" path. It's the wrong shape for tooling work
(LSP, REPL, formatter, multi-file projects) and three concrete things in the
current code already smell:

1. **`Source` is threaded through every stage.** Only the scanner truly
   consumes it; parser and interpreter only carry it for miette span
   rendering. Each stage holds something it doesn't really process.

2. **Import resolution lives inside `Interpreter`.**
   `src/interpreter/mod.rs:92-97` shows:

   ```rust
   source_dir:      Option<PathBuf>,   // for resolving relative `use`
   loaded_files:    HashSet<PathBuf>,  // canonical path dedup
   search_paths:    Vec<PathBuf>,      // -I lookup roots
   ```

   Plus a clone+merge dance to propagate `loaded_files` and `next_instance_id`
   across child interpreters when an `use` is processed. None of this is
   _interpretation_ — it's _project management_ that happens to be hosted by
   the interpreter because nothing else owns it.

3. **`Interpreter::with_search_paths(...)` is a builder on the wrong object.**
   Search paths describe the project, not one interpretation. The builder
   sits there because there is no better home.

This refactor introduces that better home.

---

## 2. Goals & non-goals

### Goals

- One explicit owner for source-file management, search paths, import
  resolution state, loaded-file dedup, and the global instance-id counter.
- Per-file caching of scan/parse/eval results, so an imported file is
  scanned/parsed/evaluated exactly once per session (matches current
  semantics, just hosted properly).
- Keep `Nexus::run` / `Nexus::run_file` signatures unchanged (or
  source-compatible — see Open Question Q4).
- Each migration step leaves the build green and all 836 tests passing.
- No semantic changes to the language. Same programs, same outputs, same
  errors.

### Non-goals (deferred)

- **Incremental invalidation.** A real LSP wants "file X changed, invalidate
  only its cache". For now caches are append-only.
- **Diagnostics-as-data.** Stages still return `Result<T, FirstError>`. The
  "collect all errors, don't bail" model is a separate, larger change.
- **Typing the source↔tokens coupling** (a wrapper type that pairs a `Source`
  with the `Tokens` scanned from it). Independent improvement; orthogonal.
- **Renaming or restructuring the pipeline stages themselves.** Scanner /
  Parser / Interpreter keep their current `new` + single-method shape.
- **Multi-threaded sessions / parallel imports.** Single-threaded for now.

---

## 3. Design overview

```
┌──────────────────────────────────────────────────────────────────────┐
│                            Session                                   │
│  ┌────────────────┐  ┌──────────────────┐  ┌─────────────────────┐   │
│  │   SourceMap    │  │ Import resolver  │  │  Per-file caches    │   │
│  │  FileId → text │  │  search_paths    │  │  tokens / ast / net │   │
│  └────────────────┘  └──────────────────┘  └─────────────────────┘   │
│  ┌────────────────┐  ┌──────────────────┐                            │
│  │ next_instance  │  │   diagnostics    │                            │
│  │      _id       │  │  (warnings)      │                            │
│  └────────────────┘  └──────────────────┘                            │
└──────────────────────────────────────────────────────────────────────┘
            │ owns / drives
            ▼
┌──────────────────────────────────────────────────────────────────────┐
│           Scanner   →   Parser   →   Interpreter                     │
│         (unchanged)   (unchanged)   (slimmer — no project state)     │
└──────────────────────────────────────────────────────────────────────┘
```

The pipeline stages stay as they are. The session owns everything they used
to carry around or fabricate internally.

---

## 4. Detailed design

### 4.1 `FileId`

Opaque, cheap to copy, stable within a session.

```rust
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub struct FileId(u32);
```

Sessions allocate `FileId`s monotonically. Used as a cache key throughout.

### 4.2 `SourceMap`

Owns the actual source strings so `Source<'_>` borrows are well-defined for
the lifetime of the session. Maps:

- `FileId → (canonical path | None, source text)`
- `canonical path → FileId` (for dedup on import)

Two ways a file enters the map:

- `load_string(code: &str, name: Option<&str>) → FileId` — inline source
  (REPL, `Nexus::run`, tests). No canonical path entry; only a synthetic
  name for diagnostics.
- `load_file(path: impl AsRef<Path>) → io::Result<FileId>` — read from disk,
  canonicalize the path, dedup against `loaded_files`.

### 4.3 `Session` — public shape

```rust
pub struct Session {
    sources: SourceMap,
    search_paths: Vec<PathBuf>,
    next_instance_id: u64,
    // caches (each entry computed lazily on first request, then memoized):
    tokens: HashMap<FileId, Tokens>,
    asts: HashMap<FileId, Ast>,
    // result of evaluating a file's top-level — what `use` merges into a parent:
    exports: HashMap<FileId, ExportBundle>,
    warnings: Vec<InterpretWarning>,
}

impl Session {
    pub fn new() -> Self;
    pub fn with_search_paths(self, paths: Vec<PathBuf>) -> Self;

    // Source loading.
    pub fn load_string(&mut self, code: &str, name: Option<&str>) -> FileId;
    pub fn load_file(&mut self, path: impl AsRef<Path>) -> io::Result<FileId>;

    // Lazy pipeline queries. Each one computes-and-caches if needed,
    // then returns a reference to the cached value.
    pub fn tokens(&mut self, f: FileId)  -> Result<&Tokens, ScanError>;
    pub fn ast(&mut self, f: FileId)     -> Result<&Ast, ParseError>;
    pub fn evaluate(&mut self, f: FileId) -> Result<Network, InterpretError>;

    // Import resolution — called by the interpreter, not by users.
    pub(crate) fn resolve_import(&self, importer: FileId, name: &str)
        -> Result<FileId, ImportError>;
    pub(crate) fn exports_of(&mut self, f: FileId)
        -> Result<&ExportBundle, InterpretError>;

    // Diagnostics.
    pub fn warnings(&self) -> &[InterpretWarning];
    pub fn into_warnings(self) -> Vec<InterpretWarning>;
}
```

`ExportBundle` is whatever the interpreter currently produces and merges from
an imported file (top-level bindings + connections + node-id assignments).

### 4.4 Slimmer `Interpreter`

After the refactor, `Interpreter` no longer owns:

- `source_dir`, `loaded_files`, `search_paths` — gone.
- `next_instance_id` — moved to `Session`; the interpreter asks the session
  to mint new IDs.
- `warnings` — appended to session's collection.

`Interpreter::new` takes a session reference (or a small handle object) so
it can call `resolve_import`, `exports_of`, `mint_node_id`, and
`push_warning` on it.

```rust
pub struct Interpreter<'a, 's> {
    session: &'s mut Session,
    file:    FileId,
    ast:     &'s Ast,           // borrowed from session cache
    env:     Environment,
    // ... per-interpretation state only (depths, stacks, member_stack, etc.) ...
}
```

`eval_use` becomes (sketch):

```rust
fn eval_use(&mut self, name: &str) -> Result<(), InterpretError> {
    let imported = self.session.resolve_import(self.file, name)?;
    let exports  = self.session.exports_of(imported)?;
    self.env.merge_global_bindings(exports);
    Ok(())
}
```

The recursive clone+merge dance disappears because the cache absorbs it.

### 4.5 `Nexus` re-implementation

The user-facing entry points become thin wrappers. `run_file` takes
`impl AsRef<Path>` — idiomatic Rust for file paths, accepts `&str`,
`String`, `&Path`, `PathBuf`, `&PathBuf`. This change is independent of
the session refactor and lands first as Step 0.

```rust
use std::path::Path;

impl Nexus {
    pub fn run(source: &str) -> Result<(Network, Vec<Warning>), NexusError> {
        let mut s = Session::new();
        let f = s.load_string(source, None);
        let net = s.evaluate(f)?;
        Ok((net, s.into_warnings().into_iter().map(Warning::from).collect()))
    }

    pub fn run_file(path: impl AsRef<Path>) -> Result<(Network, Vec<Warning>), NexusError> {
        let mut s = Session::new();
        let f = s.load_file(path)?;
        let net = s.evaluate(f)?;
        Ok((net, s.into_warnings().into_iter().map(Warning::from).collect()))
    }
}
```

Same external behavior, same error types, same warning shape.

### 4.6 Test impact analysis (why `Interpreter` can be hidden)

Inventory of every test that touches a hidden-or-hideable type today:

| Test file                    | Imports                                                              | Truly needs internals?                                                                                                                          |
| ---------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `tests/interpreter_tests.rs` | `Interpreter`, `Parser`, `Scanner`, `Source`                         | **No** — pass/fail harness that runs `.nxs` files. Only checks `is_ok()` / `is_err()`. Trivially rewritten in terms of `Nexus::run_file(path)`. |
| `tests/parser_tests.rs`      | `Parser`, `Scanner`, `Stmt`                                          | Yes — needs the AST shape (`Stmt`) to verify parses.                                                                                            |
| `tests/scanner_tests.rs`     | `Scanner`, `Tokens`                                                  | Yes — needs the token vector.                                                                                                                   |
| `tests/scanner/scanner.rs`   | `Scanner`, `TokenKind`                                               | Yes — unit-tests scanner behavior token-by-token.                                                                                               |
| `tests/scanner/token.rs`     | `Location`, `Token`, `TokenKind`                                     | Yes — unit-tests `Token`.                                                                                                                       |
| `tests/scanner/cursor.rs`    | `Location`, `scanner::cursor::Cursor`                                | Yes — unit-tests the cursor.                                                                                                                    |
| `tests/parser/cursor.rs`     | `Location`, `Token`, `TokenKind`, `Tokens`, `parser::cursor::Cursor` | Yes — unit-tests the cursor.                                                                                                                    |

Only **one** test reaches into `Interpreter`, and it doesn't actually need to.
After Step 7 rewrites `tests/interpreter_tests.rs` against `Nexus::run_file`,
`Interpreter` has zero external consumers and can be removed from the
crate-root `#[doc(hidden)] pub use`. It stays `pub` inside the (private)
`mod interpreter` so the session can construct it.

`Scanner` and `Parser` keep their `#[doc(hidden)] pub use` re-exports —
unit-test harnesses use them directly. They're already invisible in docs;
that's the right end state.

---

## 5. Migration plan

Each step is its own commit and must leave `cargo test` green. Steps are
ordered so that intermediate states are coherent — no step assumes a future
step's work.

### Step 0 — Path-typed `run_file` (independent prerequisite) ✅

Standalone improvement, not part of the session refactor — but it lands first
because Step 7 wants to call `Nexus::run_file(path)` from the migrated
`tests/interpreter_tests.rs`, and the test passes paths around as `&Path` /
`PathBuf`.

- Change `Nexus::run_file(path: &str)` to `Nexus::run_file(path: impl AsRef<Path>)`.
- No internal callers exist today (confirmed by grep), so this only affects
  the published API shape — additive (more types accepted).

**Done.** Signature updated to `impl AsRef<std::path::Path>`; build, all 836
tests, and clippy pass. Internals still use the manual scan/parse/eval chain
— they'll be replaced when the `Session` lands (Steps 3 onward). The public
shape is now final.

### Step 1 — Introduce `SourceMap` and `FileId` (no behavior change)

- Add `src/session/source_map.rs` with `FileId`, `SourceMap`.
- `SourceMap` owns source strings (likely `String` per file; revisit if the
  borrow story gets ugly).
- `Source<'_>` becomes a view borrowed from the map. Today `Source<'a>`
  already wraps `&'a str` — keep that, just lengthen its provenance.
- No callers use this yet. Pure addition, unit-tested in isolation.

**Done when:** `SourceMap` has `load_string` / `load_file` / `get(id)`, plus
unit tests for dedup and round-trip.

### Step 2 — Introduce empty `Session` shell

- Add `src/session/mod.rs` with `Session { sources: SourceMap }`.
- `Session::new()`, `Session::load_string`, `Session::load_file` delegate to
  `SourceMap`.
- Add `Session::with_search_paths`. Holds the field; nobody reads it yet.
- Re-export `Session` and `FileId` from `lib.rs` under
  `#[doc(hidden)]` for now — promote to public API only at the end (Q4).

**Done when:** `Session` exists, compiles, has trivial tests.

### Step 3 — `Session` owns `tokens()` and `ast()` (still no interpreter changes)

- Implement `Session::tokens(file)` — runs `Scanner` if not cached.
- Implement `Session::ast(file)` — calls `self.tokens(file)?` then `Parser`.
- Wire `Nexus::run` / `Nexus::run_file` to construct a `Session`, load the
  source, and continue to use `Interpreter::new(source, ast)` as today.
- Binary (`src/bin/nexus.rs`) keeps doing what it does today — direct stage
  access for its "show tokens, then AST, then result" mode. Updating the
  binary to use the session is optional and can happen later.

**Done when:** Nexus entry points go through `Session` for scan/parse;
interpreter still unchanged.

### Step 4 — Move `search_paths` and `next_instance_id` off `Interpreter`

- Add `mint_node_id()` (or similar) to `Session`. Interpreter calls it
  instead of incrementing `self.next_instance_id`.
- Move `search_paths` to `Session`. Drop `Interpreter::with_search_paths`
  (deprecated path: leave a thin shim that forwards to the session if it
  helps the binary, then delete in a later step).
- `Interpreter::new` gains a `&mut Session` parameter (or a handle struct
  that bundles `&mut Session, FileId`).

**Done when:** `Interpreter` no longer has `next_instance_id` or
`search_paths` fields.

### Step 5 — Move import resolution and `loaded_files` off `Interpreter`

- Add `Session::resolve_import(importer: FileId, name: &str) -> Result<FileId, ImportError>`.
- Add `Session::exports_of(file: FileId) -> Result<&ExportBundle, InterpretError>`.
- `exports_of` evaluates the file's top-level once and caches the result
  (bindings + connections + minted IDs).
- Cycle detection: `exports_of` tracks an in-progress set; re-entry yields
  `ImportError::Cycle`.
- `eval_use` is rewritten to call into the session. The clone+merge dance and
  child-interpreter creation disappear.
- `loaded_files` and `source_dir` removed from `Interpreter`.

**Done when:** `Interpreter` no longer owns any import-related state. All
`use`-related tests still pass.

### Step 6 — Move warnings to `Session`

- `Session::warnings` accumulates `InterpretWarning`s during evaluation.
- `Interpreter` no longer owns a `warnings` vec; it pushes to the session.
- `Session::evaluate` returns `Result<Network, InterpretError>`; warnings are
  read separately via `Session::warnings()` / `Session::into_warnings()`.
- `Nexus::run` collects warnings from the session at the end (signature
  unchanged externally).

**Done when:** Warning accumulation lives on the session, not the interpreter.

### Step 7 — Cleanup, test migration, hide `Interpreter`

- Switch `src/bin/nexus.rs` to the `Session` API (per D7). It becomes the
  canonical user of `session.tokens(f)`, `session.ast(f)`, `session.evaluate(f)`.
  The binary loses its direct `Scanner` / `Parser` / `Interpreter`
  constructions.
- Rewrite `tests/interpreter_tests.rs` to call `Nexus::run_file(path)`
  instead of stitching `Scanner` → `Parser` → `Interpreter` by hand. The
  test only checks pass/fail; the high-level entry point is a drop-in fit.
- After both of the above land, `Interpreter` has no external consumers.
  Remove it from the `#[doc(hidden)] pub use` in `src/lib.rs`. It stays
  `pub` inside the (private) `mod interpreter` so the session reaches it
  through `crate::interpreter::Interpreter` paths.
- Same treatment is **not** applied to `Scanner` / `Parser`: unit tests
  (`tests/scanner/*.rs`, `tests/parser/cursor.rs`, the harness tests)
  legitimately exercise them and keep their `#[doc(hidden)]` re-exports.
- Stop threading `Source` through stages that don't need it (parser,
  interpreter take `FileId` and get their `Source` from the session).
- Delete any leftover compatibility shims.
- Update `CLAUDE.md` and `README.md` to mention the session layer.
- Update memory.

**Done when:**

- Binary uses session, not stage objects.
- `tests/interpreter_tests.rs` uses `Nexus::run_file`.
- `Interpreter` no longer appears in `lib.rs`'s `pub use` list.
- All 836 tests pass.
- Docs updated.

---

## 6. Resolved design decisions

All open questions decided up front. Implementation follows these.

- **D1: Name → `Session`.**
  No alternative considered worth the bikeshedding cost.

- **D2: `FileId` is an opaque `u32` index.**
  `SourceMap` owns the path / text behind it; callers treat `FileId` as
  pointer-cheap and hashable.

- **D3: Source ownership: borrow-based first, `Arc<str>` only as fallback.**
  Attempt to keep `Source<'_>` borrowing from session-internal `String`s.
  If the borrow checker fights the mutable-self pipeline too hard, switch
  to `Arc<str>` per source. Decide _during_ Step 4 — that is the first step
  where the pattern is exercised under real mutation.

- **D4: `Session` is hidden during the refactor (`#[doc(hidden)]`).**
  Public API stays as it is (`Nexus::run` / `Nexus::run_file` plus the
  network/error/warning types). Promotion to public is deferred to the
  tooling work — the shape will settle first.

- **D5: Cache both ASTs and evaluated exports.**
  Caching evaluated exports is what makes the `loaded_files` clone+merge
  dance disappear; doing it half-way leaves the dance in place.

- **D6: Diagnostics stay `Result<T, FirstError>`.**
  Warnings accumulate on the session; errors still short-circuit at the
  first occurrence. A pivot to `(Option<T>, Vec<Diag>)` is a separate,
  larger change and gets its own design doc when needed.

- **D7: Binary switches to the `Session` API.**
  `src/bin/nexus.rs` becomes the canonical user of the new API
  (`session.tokens(f)`, `session.ast(f)`, `session.evaluate(f)`), which
  stress-tests its ergonomics. Stage objects are not used directly in the
  binary after Step 7.

- **D8: `Interpreter` becomes truly internal in Step 7.**
  Revised after test inventory (§4.6): only `tests/interpreter_tests.rs`
  uses `Interpreter`, and it's a pass/fail harness that rewrites trivially
  against `Nexus::run_file`. After Step 7's migration, `Interpreter` has
  no external consumers and is removed from `lib.rs`'s `pub use`. It
  remains `pub` inside the private `mod interpreter` so the session can
  construct it. Goes further than the previous `#[doc(hidden)]` plan:
  the symbol simply isn't reachable from outside the crate anymore.

---

## 7. Risks

- **Borrow-checker pain at the `Session` ↔ `Interpreter` boundary.** Mutable
  reference to the session held by the interpreter for its whole evaluation
  blocks other session uses. Likely workaround: split mutable concerns
  (`sources` vs `caches` vs `warnings`) into separate sub-structs reachable
  through methods, so we borrow narrowly. Worst case: `RefCell`/`Cell` on
  hot fields. Detect early in Step 4 — that's where mutation patterns matter.
- **Cycle in `use` imports.** Today the interpreter detects re-entry via the
  `loaded_files` set being checked-before-add. The session needs explicit
  in-progress tracking distinct from completed-cache. Easy if planned;
  invisible bug if not.
- **`NodeId`/`GroupId` stability across re-evaluation.** Cached exports
  contain IDs minted at first evaluation. Importing the same file from two
  importers produces identical IDs in both — which matches current
  behavior, but worth a regression test that pins this down.
- **Public-API promises.** Going `pub` on `Session` too early commits us to
  a shape we haven't validated. Q4's default (start hidden) mitigates this.

---

## 8. Implementation log

Filled in as work proceeds. Each entry: date, step, what changed, any
deviation from the plan, any new question that surfaced.

### 2026-05-26 — Step 0: path-typed `run_file`

- `src/lib.rs`: `Nexus::run_file(path: &str)` → `Nexus::run_file(path: impl AsRef<std::path::Path>)`.
  Body unchanged in spirit — reads the file, converts path to a display
  string for `Source::new_with_name`, then runs scan/parse/eval. Internals
  to be replaced by `Session` calls in later steps.
- No internal callers existed (confirmed by grep), so no other code edits
  needed.
- Build, all 836 tests, and `cargo clippy` clean.
- No deviation from plan, no new questions.

---

## 9. Decision index (for quick reference)

| #   | Question                                       | Decision                          |
| --- | ---------------------------------------------- | --------------------------------- |
| D1  | Name of the object                             | `Session`                         |
| D2  | `FileId` representation                        | opaque `u32`                      |
| D3  | Source ownership / lifetimes                   | borrow-first; `Arc<str>` fallback |
| D4  | Is `Session` part of the public API?           | hidden during refactor            |
| D5  | Caching: ASTs only, or also evaluated exports? | both                              |
| D6  | Diagnostics model: keep `Result` or pivot      | keep `Result`                     |
| D7  | Binary uses session or stages directly?        | session                           |
| D8  | `Interpreter` visibility after refactor        | made truly internal in Step 7     |

## 10. Future work (out of scope here, captured so it isn't lost)

- **Promote `Session` to public API** when tooling work starts (LSP /
  formatter / REPL). The shape should have settled by then.
- **Diagnostics-as-data** (`(Option<T>, Vec<Diag>)` per stage) for partial
  results an LSP can consume even when a file has parse errors.
- **Typed source↔tokens coupling** — wrap `Source` + `Tokens` so the
  invariant ("these tokens were scanned from this source") is enforced by
  the type system.
- **Incremental invalidation** in the session caches — needed once an LSP
  starts re-querying on every keystroke.
