# Story 8C Spec: Realize Backend-Backed Query/Index Parity

Validated on 2026-04-01 against the current `megalib` tree, the Rust graph `project`, the upstream C++ graph `sdk`, and the sibling upstream SDK at `../sdk`.

This document refines Story 8C from `agents/outputs/architectural_parity_epic.md` into an implementation-ready planning story. Its job is to close the parity gap left intentionally open by Story 8: Story 8 establishes the runtime-owned query substrate, while Story 8C makes backend-backed query/index realization a first-class owned slice instead of an implied future hook.

Companion artifacts:

- `agents/outputs/architectural_parity_epic.md`
- `agents/outputs/architectural_parity_story_4b_production_persistence_backend.md`
- `agents/outputs/architectural_parity_story_4c_production_tree_cache_hardening.md`
- `agents/outputs/architectural_parity_story_8_query_index_runtime.md`
- `agents/outputs/architectural_parity_story_12_validation_harness.md`

---

## Status

Not started.

Current implementation status on 2026-04-01:

- current Rust browse/query behavior is still direct cached-tree traversal
- Story 8 is the planned runtime substrate story and intentionally stops at an in-memory evaluator plus backend hooks
- there is no separate owned story yet for backend-backed query/index realization
- the epic still tracks DB-backed query/search parity as an architectural target

This means Story 8 alone is not enough to close the full query/index row honestly.

---

## Story Goal

Extend the Story 8 substrate so `megalib` has an explicit backend-backed query/index realization under the durable node-cache layer:

- backend-owned query/index lifecycle becomes explicit
- query/index rebuild and recycle behavior ties to the same durable tree boundaries as the node cache
- the epic’s DB-backed query/index target gets a real owning story

---

## Why This Story Exists

The upstream SDK does not stop at an in-memory browse seam. Its query/index behavior lives with `NodeManager` and the DB-backed node tables:

- query/search interfaces in `../sdk/include/mega/nodemanager.h`
- backend query/index support in `../sdk/include/mega/db.h`
- query/index lifecycle in `../sdk/src/nodemanager.cpp`
- SQLite-backed index creation/drop behavior in `../sdk/src/db/sqlite.cpp`

Story 8 intentionally stops short of that because it is the substrate story:

- one runtime-owned query model
- one in-memory evaluator
- hook points for later backend acceleration

That is correct sequencing, but not complete parity. Story 8C exists so “later backend work” is not left unowned.

---

## Scope

In scope:

- define the backend-owned query/index hook contract below Story 8’s runtime API
- add one real backend-backed query/index realization
- tie backend index lifecycle to durable node-cache restore, refresh, AP commit, and recycle events
- add restart, recycle, and backend query-behavior tests

Out of scope:

- public search API widening
- changing the authoritative tree snapshot model
- generic persistence-SPI redesign
- making query indexes a second source of truth
- solving every possible upstream search/filter feature in one slice

This story is successful if the query row no longer relies on “hooks exist” as its end state.

---

## Story 4B, Story 4C, And Story 8 Constraints

Story 8C must preserve these existing decisions:

- Story 8 owns the public-compatible runtime query substrate
- Story 4B owns the production backend root and lifecycle baseline
- Story 4C owns production-backed durable tree/cache correctness
- the tree snapshot remains authoritative; query/index structures are derived
- backend query/index work must stay below the runtime query API and above raw backend storage details

If implementation pressure suggests widening `src/session/runtime/persistence.rs` into a generic query API or making index state authoritative over the tree snapshot, the design is wrong.

---

## SDK Parity Target

Story 8C should align with the upstream SDK in these ways:

1. Backend-owned query/index realization exists under the node-cache layer.
2. Index lifecycle is tied to durable node-cache lifecycle, including rebuild and recycle behavior.
3. Query/index state is derived and discardable, not authoritative account truth.
4. Restart behavior over the real backend path is validated rather than inferred.

Rust should stay idiomatic:

- do not clone `NodeManager` internals mechanically
- do keep one explicit backend-owned query/index owner
- do keep Story 8’s runtime query API stable while deepening the backing implementation

---

## Design Decisions

### Decision 1. Story 8 owns the runtime API; Story 8C owns the backend realization

Why:

- Story 8 already established the correct runtime seam
- the missing piece is backend realization, not another public-compatible query redesign

Consequence:

- Story 8C must consume Story 8’s hook seam rather than inventing a second query API

### Decision 2. Backend query/index state is derived and recyclable

Why:

- upstream index lifecycle is coupled to DB recreate/drop flows
- query/index state should never outrank the durable tree snapshot

Consequence:

- index corruption or incompatibility should trigger rebuild/recycle behavior
- restore fallback should preserve the authoritative tree snapshot while rebuilding or discarding derived query/index state

### Decision 3. Lifecycle must match the durable tree boundaries

Why:

- Story 4 and Story 4C already define the coherent restore/refresh/AP boundaries
- backend query/index realization only remains correct if it follows those same boundaries

Consequence:

- Story 8C rebuild or invalidation must happen on:
  - startup restore
  - successful refresh replacement
  - successful durable AP commit
  - backend recycle or incompatible schema reset

### Decision 4. Backend realization belongs with the node-cache backend, not the generic persistence SPI

Why:

- the epic and Story 8 already treat query/index work as a node-cache concern
- generic persistence SPI broadening would blur ownership

Consequence:

- Story 8C may add backend-facing modules or sibling helpers
- Story 8C should not turn `PersistenceRuntime` into the public or generic query owner

### Decision 5. The first backend realization may be narrow, but it must be real

Why:

- the point of Story 8C is to stop relying on “hooks only”
- a small real backend realization is better than a broad speculative design

Consequence:

- the first live slice may target a limited query subset such as:
  - child lookup
  - recent files
  - text search over already-owned fields
- but it must execute through a backend-owned realization, not only through in-memory scans

---

## Recommended Rust Shape

One acceptable direction is:

```rust
// illustrative only

pub(crate) trait QueryBackend {
    fn rebuild_from_tree(&mut self, tree: &PersistedTreeState) -> Result<()>;
    fn drop_indexes(&mut self) -> Result<()>;
}

pub(crate) struct SqliteQueryBackend {
    // backend-owned query/index state
}
```

The exact types may differ. The architectural requirements are:

- Story 8 query runtime remains the entrypoint
- one backend-owned query/index realization exists
- rebuild, drop, and recycle rules are explicit
- derived backend state never becomes authoritative over the tree snapshot

---

## Affected Modules

Expected primary touch points:

- `src/fs/runtime/query.rs`
- node-cache/backend modules introduced for query/index realization
- backend lifecycle modules from Stories 4B and 4C as needed

Possible supporting modules:

- `src/session/core.rs`
- `src/session/action_packets.rs`
- tests alongside the query and backend modules

---

## Agent-Sized Tasks

### Task 8C.1

Define the backend-owned query/index contract below Story 8’s runtime seam.

### Task 8C.2

Implement one real backend-backed query/index realization.

### Task 8C.3

Wire rebuild, recycle, and invalidation behavior to the durable tree/backend lifecycle.

### Task 8C.4

Add focused restart, recycle, and query-behavior coverage on the real backend path.

---

## Acceptance Criteria

Story 8C is complete when:

1. backend-backed query/index realization has an explicit owning story and implementation path
2. Story 8’s query runtime can rely on a real backend-owned realization rather than only in-memory scans for parity claims
3. backend query/index state is explicitly derived and recyclable
4. restart, refresh, AP commit, and recycle behavior are validated against the real backend path
5. Story 8’s runtime API remains the substrate while Story 8C closes the DB-backed parity gap

---

## Verification Requirements

Because this is a future Rust-touching story, each implementation slice must end with:

```bash
cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all
```

Focused coverage should include:

- backend query/index rebuild after restore
- backend recycle or index-drop/recreate behavior
- restart behavior on the real backend path
- fallback when derived query/index state is malformed or incompatible

---

## Relationship To Later Stories

Story 8C is the backend-realization follow-on to Story 8.

Later work should consume it like this:

- Story 12 should add backend query/index rebuild, recycle, and restart scenarios
- later public search work should expose public APIs over Story 8’s runtime substrate without reintroducing direct backend coupling
- later desktop stories may benefit from the accelerated backend path, but they should continue to depend on the stable Story 8 query API rather than backend details

The key architectural rule is:

- Story 8 defines the runtime query substrate
- Story 8C makes the backend realization parity claim true
