# iqdb-filter v0.3.0 — Strategy Selection

**The vocabulary becomes actionable.** v0.2.0 shipped the canonical evaluator and named the strategy shapes; v0.3.0 makes them usable. The evaluator gains two scan helpers that apply a strategy, a structural selectivity estimate, and a tunable selector that picks `PreFilter` or `PostFilter` for you — all without an inverted index, and all shaped to how `iqdb-flat` actually filters today.

## What is iqdb-filter?

The metadata-filtering layer of the iQDB vector database. Given a `Filter` and a record's `Metadata`, it answers "does this record match?" — and now also "how should an index *apply* this filter relative to its distance scan?". The crate owns the filter semantics so every index above it shares one verified implementation.

## What's new in 0.3.0

### Scan helpers: `prefilter` and `postfilter`

Two lazy, allocation-free iterator adapters over a stream of `(key, metadata)` pairs. They share the per-row test; the difference is purely where in a search pipeline they run.

```rust
use iqdb_filter::FilterEvaluator;
use iqdb_types::{Filter, Metadata, Value};

let evaluator = FilterEvaluator::new(Filter::gt("year", Value::Int(2000)))
    .expect("valid filter");

let a: Metadata = [("year".to_string(), Value::Int(2026))].into_iter().collect();
let b: Metadata = [("year".to_string(), Value::Int(1999))].into_iter().collect();
let rows = [(0_usize, Some(&a)), (1, Some(&b)), (2, None)];

// PreFilter: keep matching candidate keys before scoring.
let kept: Vec<usize> = evaluator.prefilter(rows).collect();
assert_eq!(kept, [0]);
```

`postfilter` is the symmetric post-scoring form; because it is lazy, a caller refilling a top-`k` result set can chain `.take(k)` and stop as soon as `k` survivors are found.

### Structural selectivity estimation

`estimate_selectivity(&FilterEvaluator) -> f64` returns a best-effort estimate, in `[0.0, 1.0]`, of the fraction of records a filter passes — derived from the filter's **shape**, not from any data. Equality is narrow, ranges wider, `Neq` broad; `And` multiplies, `Or` unions, `Not` complements. It takes a validated evaluator, so the walk is depth-bounded and cannot overflow on adversarial input.

```rust
use iqdb_filter::{FilterEvaluator, estimate_selectivity};
use iqdb_types::{Filter, Value};

let eq = FilterEvaluator::new(Filter::eq("status", Value::Int(1))).expect("valid");
let neq = FilterEvaluator::new(Filter::neq("status", Value::Int(1))).expect("valid");
assert!(estimate_selectivity(&eq) < estimate_selectivity(&neq));
```

### Automatic strategy selection (Tier 1 + Tier 2)

`choose_strategy` resolves a concrete `FilterStrategy` from the estimate using the default threshold; `StrategySelector` is the immutable, tunable builder for the cutoff. A narrow predicate resolves to `PreFilter` (skip the distance work for rejected rows); a broad one to `PostFilter`. The selector never returns `Auto` (it is the thing that resolves it) or `InFilter` (which needs graph-traversal co-design).

```rust
use iqdb_filter::{FilterEvaluator, FilterStrategy, StrategySelector, choose_strategy};
use iqdb_types::{Filter, Value};

let narrow = FilterEvaluator::new(Filter::eq("id", Value::Int(7))).expect("valid");
assert_eq!(choose_strategy(&narrow), FilterStrategy::PreFilter);

// Tune the cutoff: force pre-filtering even for a broad predicate.
let broad = FilterEvaluator::new(Filter::neq("id", Value::Int(7))).expect("valid");
let always_pre = StrategySelector::new().with_prefilter_threshold(1.0);
assert_eq!(always_pre.choose(&broad), FilterStrategy::PreFilter);
```

`DEFAULT_PREFILTER_THRESHOLD` (`0.5`) is the documented default and a tunable threshold is exposed precisely because the structural estimate is coarse.

## Performance

All paths are allocation-free. On the reference machine:

| Path | Time |
|------|------|
| `estimate_selectivity` (compound filter) | ~10 ns |
| `choose_strategy` (compound filter) | ~11 ns |
| `prefilter` (1,000 rows, 50% match) | ~14.8 µs (~15 ns/row) |
| `evaluate` (per row, with metadata) | ~19 ns |

Numbers are from the `criterion` bench (`cargo bench`).

## What's deferred (and why)

The selectivity estimate here is **structural** by design. An **index-backed** estimate (per-field histograms and sketches) and the inverted `MetadataIndex` it would read remain deferred, along with `InFilter` pushdown into graph traversal. Each needs a real approximate-index consumer (`iqdb-hnsw` / `iqdb-ivf`) to validate thresholds and recall before it can be built honestly. The tunable threshold on `StrategySelector` is the seam an index-backed estimator slots behind later. See `dev/ROADMAP.md`.

## Breaking changes

**Pre-1.0 additive release.** Everything here is new public surface; nothing from v0.2.0 changed shape. `FilterStrategy::Auto` is now resolved by the selector rather than being inert vocabulary.

## Verification

The new surface is covered by unit tests (selectivity ordering and bounds, threshold clamping, prefilter/postfilter behaviour), property tests (selectivity is always a probability; `prefilter` yields exactly the manually-filtered subset; the selector never returns `Auto`/`InFilter`), and benchmarks. The same gates run across the CI matrix (Linux, macOS, Windows) on stable and the 1.87 MSRV:

```bash
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
```

MSRV: Rust 1.87.

## What's next

- **Index phase.** When an approximate index starts honouring filters: the opt-in inverted `MetadataIndex`, an index-backed selectivity estimate, and `InFilter` pushdown into graph traversal.

## Installation

```toml
[dependencies]
iqdb-filter = "0.3"
```

## Documentation

- [README](https://github.com/jamesgober/iqdb-filter/blob/main/README.md)
- [API reference](https://github.com/jamesgober/iqdb-filter/blob/main/docs/API.md)
- [ROADMAP](https://github.com/jamesgober/iqdb-filter/blob/main/dev/ROADMAP.md)
- [Standards (REPS)](https://github.com/jamesgober/iqdb-filter/blob/main/REPS.md)
- [CHANGELOG](https://github.com/jamesgober/iqdb-filter/blob/main/CHANGELOG.md)

---

**Full diff:** [`v0.2.0...v0.3.0`](https://github.com/jamesgober/iqdb-filter/compare/v0.2.0...v0.3.0).
**Changelog:** [`CHANGELOG.md`](https://github.com/jamesgober/iqdb-filter/blob/main/CHANGELOG.md).
