# skade — roadmap / "the list"

Forward-looking work. skade today is a correct, ergonomic, *fast-enough* layer
over iceberg-rust + the pure-Rust nornir-catalog (`RedbCatalog`). It is **not yet**
the maximal zero-copy / SIMD / all-cores engine — these items close that gap.

## 1. Kill the C deps → route the codec path through nordisk crates (incl. #3 zero-copy)

The win here is **not** rewriting our own read code (our crates are already
zero-copy) and **not** patching upstream's logic — it's making the iceberg/
parquet/avro codec path **use our crates** instead of the stock C-backed ones.

Status today (measured via `cargo tree`):

| C dep | Enters via | Default build (0.3.0) |
|---|---|---|
| `liblzma-sys` (xz) | datafusion → async-compression (the `sql` feature) | **gone** — `sql` is opt-in as of 0.3.0 |
| `zstd-sys` (libzstd) | iceberg-rust → **apache-avro** (manifest codec) | **still present** — rides in with iceberg itself |

Plan: **patch it away.** Replace the stock C-backed codecs in the iceberg/
datafusion path with the nordisk implementations:
- **parallelized lzma** — `znippy-zoomies/lzip-parallel` (multi-core xz/lzma).
- **zero-copy lzma + zstd** — `zstd-sys-rs` (zero-copy zstd API) and the
  zero-copy lzma path, so the Avro/Parquet compression in the iceberg layer
  stops copying and stops pulling stock C crates we don't control.

This needs upstream/patch work on `iceberg-rust` (apache-avro codec) and
`datafusion` (async-compression) — a `[patch.crates-io]` to nordisk forks, the
same mechanism already used for the parallel iceberg-datafusion scan.

## 2. Dogfood skade on the TPC-H benches

nornir-catalog's TPC-H suite (`bench/tpch_shared.rs`) wires DataFusion to
`RedbCatalog` via `iceberg_datafusion::IcebergCatalogProvider` **directly** — it
does not go through skade (the bench crate doesn't even depend on skade). Route
it through skade's `sql` feature instead, so the SQL surface skade ships is the
one that's benchmarked, and skade gets real TPC-H coverage.

## Done
- 0.2.0 — identity-partitioned writes (`create_partitioned_table` + per-commit
  `partition_key_for`).
- 0.3.0 — `sql` opt-in (lean default, drops `liblzma-sys` + datafusion);
  criterion throughput bench (`cargo bench -p skade`).
- 0.4.0 — **gatling multi-core writes**: `ingest_parallel` (+ `Table::ingest_parallel`)
  fans Parquet encoding across all cores via rayon, one sequential writer commits
  every N files; partition-correct. Measured ~**3.8×** on the encode-heavy path
  (1 core 123ms → all cores 32ms; `ingest_parallel_scaling` bench).
