# iqdb-build v0.3.0 — Parallel sharded construction

**The bulk path goes wide.** v0.3.0 adds the feature the crate is named for:
split a large input into shards and build them concurrently across every CPU
core. A single index cannot be built in parallel — every `insert` takes
`&mut self` — so the bulk path is to construct independent sub-indexes side by
side. Combining them into one index is the next phase; the shards this release
produces are already useful to the engine, whose storage is itself sharded.

## What is iqdb-build?

The bulk-construction layer of the iQDB vector database. It is **generic over**
`iqdb_index::Index`, so the same builder constructs a flat, HNSW, or IVF index —
or a backend of your own — without ever naming a concrete type. It sits above
`iqdb-types` and `iqdb-index` and is consumed by `iqdb` for bulk ingestion.

## What's new in 0.3.0

- **`IndexBuilder::build_parallel(items)`** — splits the input into contiguous,
  near-equal shards and constructs one sub-index per shard on rayon's
  work-stealing pool. Each shard runs `Index::new` with the builder's
  `dim`/`metric`/`config` and inserts its slice through `insert_batch`. The
  returned `Vec<I>` preserves shard order. Wall-clock build time drops roughly
  linearly with cores, because the shards share nothing.
- **`IndexBuilder::with_shards(n)`** — an immutable, `#[must_use]` setter for the
  shard count. The default is one shard per available CPU; `0` is treated as `1`,
  and the effective count is clamped to `1..=items.len()` so there are no empty
  shards (empty input yields a single empty index).
- **`IndexBuilder::shards()`** — read back the configured count (`None` = auto).
- **`rayon`** is now a core dependency. The crate's whole purpose is the parallel
  bulk path, so it is wired in directly rather than hidden behind a feature.

The sequential `build` / `build_into` surface from 0.2.0 is unchanged.

## How to use it

```rust
use std::sync::Arc;
use iqdb_build::IndexBuilder;
use iqdb_types::{DistanceMetric, VectorId};

let items: Vec<_> = (0u64..1_000_000)
    .map(|i| (VectorId::from(i), Arc::from(embedding(i).as_slice()), None))
    .collect();

// Auto shards: one per CPU. `MyIndex: iqdb_index::Index`.
let shards: Vec<MyIndex> = IndexBuilder::new(768, DistanceMetric::Cosine)
    .build_parallel(items)?;

// Or pin the shard count to match the engine's layout:
let shards: Vec<MyIndex> = IndexBuilder::new(768, DistanceMetric::Cosine)
    .with_shards(8)
    .build_parallel(items)?;
```

## Notes & semantics

- **Per-shard duplicate detection.** Two items with the same id in *different*
  shards are not caught by `build_parallel` (they would collide at merge time).
  Within a shard, a repeated id errors with `Duplicate`, exactly as in `build`.
- **Bound.** `build_parallel` adds `I::Config: Send + Sync` so the config can be
  shared across worker threads; the index type itself is already `Send + Sync`
  via the `IndexCore` supertrait.

## Breaking changes

None. The release is purely additive over 0.2.0. (`rayon` joins the dependency
tree as a core dependency.)

## Verification

```bash
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features        # integration + property + doctests
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo bench --no-run             # sequential + parallel criterion groups compile
```

Tested across the CI matrix (Linux, macOS, Windows) on stable and the 1.87 MSRV.
The new path is property-tested: sharded build preserves every item, and the
shard count obeys the `1..=len` clamp for any requested count.

## What's next

- **v0.4.0 — index merging (`Mergeable`) + progress reporting + feature freeze.**
  Combine the sub-indexes from `build_parallel` into a single index, with
  optional progress callbacks for long-running builds. Feature freeze is declared
  at the close of this phase.
- **v0.5.0 — concurrent-build correctness + API freeze.** `loom`-checked
  concurrent construction; the public surface is frozen and recorded in the
  ROADMAP.

## Installation

```toml
[dependencies]
iqdb-build = "0.3"
```

## Documentation

- [README](https://github.com/jamesgober/iqdb-build/blob/main/README.md)
- [API reference](https://github.com/jamesgober/iqdb-build/blob/main/docs/API.md)
- [ROADMAP](https://github.com/jamesgober/iqdb-build/blob/main/dev/ROADMAP.md)
- [Standards (REPS)](https://github.com/jamesgober/iqdb-build/blob/main/REPS.md)
- [CHANGELOG](https://github.com/jamesgober/iqdb-build/blob/main/CHANGELOG.md)

---

**Changelog:** [`CHANGELOG.md`](https://github.com/jamesgober/iqdb-build/blob/main/CHANGELOG.md).
