# nidus

A small, pure-Rust **embeddable vector store**. Brute-force cosine search over a
single append-only directory, with typed metadata filters and many logical
collections sharing one embedding space. No FFI, no C, no SQL, no query engine.

> _nidus_ (Latin, "nest") — a small place where things are kept safe.

## Why it exists

nidus is the local storage leg for semantic-search and indexing tools: chunk some
source → embed each chunk → store the vectors + metadata → ask for nearest
neighbours. The obvious off-the-shelf options fail the **build-and-ship** test, not
the functionality test:

- **DuckDB** (via `libduckdb-sys`) bundles a large C++ source tree and compiles it
  from scratch — multi-minute cold builds, a required C++ toolchain, a bloated
  binary, and FFI that can't run under Miri. A vector workload uses ~1% of it.
- **LanceDB** is "written in Rust" yet still takes ~10 minutes to compile, because
  it drags in Arrow + DataFusion (a full SQL engine) + a columnar format — hundreds
  of crates to do `ORDER BY distance LIMIT k`.

The workload is a *vector store, not a database*. nidus is that store and nothing
more, so it **compiles in seconds** and embeds as a normal Rust dependency.

### The constraints are the product

- **Pure-Rust dependencies only** — never a crate that compiles C or links a native
  library (no `*-sys`, no bundled C/C++).
- **Zero FFI, zero `unsafe`** in our code (`#![forbid(unsafe_code)]`).
- **No C to compile** — `cargo build` is just rustc.
- **Fully Miri-checkable** — including the file IO.

## Quick start

```toml
[dependencies]
nidus = "0.1"
```

```rust
use std::collections::BTreeMap;
use nidus::{Nidus, Config, Record, SearchOpts, Scope, Value, Filter, Predicate};

// Open (or create) a store. The directory is always the caller's choice;
// the dimension is pinned for the life of the store.
let mut db = Nidus::open(Config::new("/path/to/store", 4))?;
db.create_collection("code")?;

// Index some records: id + embedding + arbitrary typed metadata.
let mut attrs = BTreeMap::new();
attrs.insert("path".into(), Value::Str("src/auth/login.rs".into()));
db.upsert("code", &[Record { id: "a".into(), vector: vec![1.0, 0.0, 0.0, 0.0], attrs }])?;

// Nearest neighbours (cosine), top-k.
let hits = db.search("code", &[1.0, 0.0, 0.0, 0.0], &SearchOpts { top_k: 5, ..Default::default() })?;
for h in &hits {
    println!("{:.3}  [{}] {}", h.score, h.collection, h.id);
}

// Search the whole store at once, with a metadata filter + score floor.
let opts = SearchOpts {
    top_k: 10,
    filter: Filter(vec![Predicate::Glob("path".into(), "src/auth/*".into())]),
    min_score: Some(0.5),
};
let hits = db.search(Scope::All, &[1.0, 0.0, 0.0, 0.0], &opts)?;
# anyhow::Ok(())
```

See [`examples/demo.rs`](examples/demo.rs) for an end-to-end run (`cargo run
--example demo`).

## What it does

- **Exact brute-force cosine** — 100% recall, fast at the target scale (≤ a few
  million vectors, comfortably in RAM). Vectors are unit-normalized on insert, so a
  score is plain cosine similarity in `[-1, 1]`.
- **Scoped search** — query one collection, a subset, or the **whole store** in one
  call, merged into a single ranking. Sound because every collection shares one
  embedding space (one pinned dimension).
- **Typed metadata + filters** — attach `Str`/`Int`/`Bool`/`List`/`Null` attributes
  and filter with `Eq` / `Glob` / `In` predicates before scoring.
- **Idempotent upserts** by caller-supplied id; `delete`, `delete_where`, per-
  collection metadata.
- **Crash-safe & durable** — an append-only flat-`f32` `data` segment plus a framed,
  CRC-checked op `log` (the commit record). A crash loses at most the in-flight
  batch; a torn tail is recovered on open. Cross-process readers get a consistent,
  lock-free snapshot (`OpenMode::ReadOnly`).
- **Synchronous, runtime-agnostic** — the hot path is CPU-bound, so there's no async
  core to lock you into a runtime. `Arc<RwLock<Nidus>>` gives concurrent searchers +
  one writer; async callers bridge with `spawn_blocking`.

## On-disk layout

A store is a directory:

```
<dir>/
  data    append-only, fixed-stride, row-major f32 matrix (header pins dimension)
  log     append-only framed op stream: [len][bincode(Op)][crc32] — the commit record
  lock    O_EXCL writer-exclusion lock file
```

`open` reads `data` into RAM and replays `log` into an in-RAM index
(`collection → { id → (row, attrs) }`). Search never touches disk.

## Configuration

```rust
use std::time::Duration;
use nidus::{Config, Fsync, OpenMode};

let cfg = Config::new("/path/to/store", 768)
    .fsync(Fsync::PerBatch)          // durability granularity (default)
    .open_mode(OpenMode::ReadWrite)  // ReadOnly = no lock, search-only
    .auto_compact(Some(0.5))         // compact on open above this dead-row ratio
    .lock_ttl(Duration::from_secs(60));
```

The store **location is always the caller's choice** — nidus contributes no path
defaults, env vars, or hidden directories.

## Development

```bash
just test    # all tests
just ci       # fmt-check + clippy (-D warnings) + test
just miri     # undefined-behavior check (nightly; pure-Rust ⇒ runs end to end)
just demo     # the end-to-end example
just deps     # the dependency tree (stays short, all pure Rust)
```

Rust 1.96+ (pinned via `rust-toolchain.toml`), edition 2024.

## Design

The full design — data model, on-disk format, durability/concurrency model, and the
deferred seams (mmap, ANN/HNSW, scalar quantization, a lightweight server) — lives in
[`SPEC.md`](SPEC.md). Each module also carries its own contract in `src/<module>/SPEC.md`.

## License

[MIT](LICENSE)
