# iqdb-build v0.4.0 — Merge, the full pipeline, and feature freeze

**The thesis completes.** `iqdb-build` exists to split a large input into chunks,
build sub-indexes in parallel, and *merge them*. v0.3.0 delivered the parallel
build; v0.4.0 adds the merge — and with it the one-call pipeline that is the
crate's whole reason for being. It also adds progress reporting and declares the
feature freeze: the public surface is complete, and everything from here to 1.0
is hardening, not new API.

## What is iqdb-build?

The bulk-construction layer of the iQDB vector database. It is **generic over**
`iqdb_index::Index`, so the same builder constructs and merges a flat, HNSW, or
IVF index — or a backend of your own — without ever naming a concrete type. It
sits above `iqdb-types` and `iqdb-index` and is consumed by `iqdb` for bulk
ingestion.

## What's new in 0.4.0

- **`Mergeable`** — the trait a backend implements to say how it absorbs another
  instance of itself: `fn merge(&mut self, other: Self) -> Result<()>`. The
  *mechanism* is the backend's choice and differs per family — a flat index
  appends rows, an IVF index extends its posting lists, a graph index re-runs its
  boundary heuristics — which is exactly why it lives behind a trait rather than
  the read-only `IndexCore` surface. The trait fixes only the observable result:
  a set union, with cross-input id collisions surfacing as `Duplicate`. (It lives
  in `iqdb-build` rather than the frozen-at-1.0 `iqdb-index`.)
- **`merge(indexes: Vec<I>)`** — fold sub-indexes into one, returning `Ok(None)`
  for empty input. The natural companion to `build_parallel`.
- **`IndexBuilder::build_merged(items)`** — the full bulk pipeline,
  *split → build in parallel → merge*, in a single call for any `I: Mergeable`.
  It honours the shard count and any progress callback.
- **`IndexBuilder::on_progress` + `BuildProgress`** — register a `Send + Sync`
  callback invoked as each shard finishes building, for progress bars on
  long-running builds.

## How to use it

```rust
use std::sync::Arc;
use iqdb_build::{IndexBuilder, Mergeable, merge};
use iqdb_types::{DistanceMetric, VectorId};

// The whole pipeline in one call (MyIndex: Mergeable).
let index: MyIndex = IndexBuilder::new(768, DistanceMetric::Cosine)
    .with_shards(8)
    .on_progress(|p| eprintln!("built {}/{}", p.shards_completed, p.shards_total))
    .build_merged(items)?;

// Or keep the stages separate — the engine's storage is itself sharded.
let shards: Vec<MyIndex> = IndexBuilder::new(768, DistanceMetric::Cosine)
    .build_parallel(items)?;
let one = merge(shards)?;        // Option<MyIndex>
```

Implementing `Mergeable` for a backend is small — say how to absorb another
instance:

```rust
impl Mergeable for MyIndex {
    fn merge(&mut self, other: Self) -> Result<()> {
        if other.dim() != self.dim() || other.metric() != self.metric() {
            return Err(IqdbError::InvalidConfig { reason: "merge shape mismatch" });
        }
        // ... absorb other's contents (append / re-index / rebuild) ...
        Ok(())
    }
}
```

## Semantics & notes

- **Cross-shard duplicates surface at merge.** The per-shard parallel build
  cannot see an id that appears in two different shards; the collision is caught
  when the shards merge, as `Duplicate`. `build_merged` reports it for you.
- **Merge is not transactional.** On error the accumulator may be left partially
  merged; merge into a clone if you need the original back. This is documented on
  the trait.
- **Progress is best-effort ordering.** Shards finish concurrently, so snapshots
  may arrive on different threads; treat `shards_completed` as a monotonic
  high-water count, not a strict sequence.

## Feature freeze

The public surface is now complete: `build` / `build_into`, `build_parallel`,
`build_merged`, `Mergeable` / `merge`, and `on_progress` / `BuildProgress`. No new
API lands before 1.0. The remaining work is the `loom` concurrency-model pass over
the parallel path (0.5) and stabilization.

## Breaking changes

None. The release is purely additive over 0.3.0.

## Verification

```bash
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features        # integration + property + doctests (run repeatedly; the progress paths are concurrent)
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo bench --no-run
```

Tested across the CI matrix (Linux, macOS, Windows) on stable and the 1.87 MSRV.
A property test asserts `build_merged` equals sequential `build` in size for any
shard count and input.

## What's next

- **v0.5.0 — concurrent-build correctness + API freeze.** `loom` model checks for
  the parallel path; the public API is frozen and recorded in the ROADMAP.
- **0.6 → 0.9** — integration against real consumers, beta hardening, RC polish.
- **1.0.0** — stable; API frozen until 2.0.

## Installation

```toml
[dependencies]
iqdb-build = "0.4"
```

## Documentation

- [README](https://github.com/jamesgober/iqdb-build/blob/main/README.md)
- [API reference](https://github.com/jamesgober/iqdb-build/blob/main/docs/API.md)
- [ROADMAP](https://github.com/jamesgober/iqdb-build/blob/main/dev/ROADMAP.md)
- [Standards (REPS)](https://github.com/jamesgober/iqdb-build/blob/main/REPS.md)
- [CHANGELOG](https://github.com/jamesgober/iqdb-build/blob/main/CHANGELOG.md)

---

**Changelog:** [`CHANGELOG.md`](https://github.com/jamesgober/iqdb-build/blob/main/CHANGELOG.md).
