# skade — performance analysis

Goal: make skade *uber fast* on every read/write path. This is the deep dive —
where the time actually goes, what's been optimised (with measured numbers), and
the prioritised levers left. Benchmarks: `benches/throughput.rs`
(`cargo bench -p skade`), all numbers on `oden` (32-core).

## Hot paths (where the time goes)

| path | cost breakdown |
|---|---|
| `open` | redb open + catalog load + ensure-namespace. One-time per process. |
| `append` (1 batch) | **recast** → ParquetWriter **encode** (1 core) → DataFileWriter → `fast_append` **commit (fsync)**. Small batches are *commit/fsync-bound*. |
| `ingest` (sequential) | loops `append` per group — single-core encode, one fsync per K batches. |
| `ingest_parallel` | rayon **all-core encode** → one **sequential writer** + commit (gatling topology). Encode-bound work fans out; writer is ordered. |
| `read` (`read_all`) | iceberg scan → `to_arrow` stream → `try_collect` — **materialises every column of every row** (no projection/pushdown at the skade API). |
| `recast` | was a **full columnar copy every write**; now zero-copy on match (see below). |
| `sql` | DataFusion over the parallel-scan iceberg-datafusion (vendored). |

## Done — measured

- **Zero-copy `recast`** — cast only type-mismatched columns; matching columns
  pass through as `Arc` clones (re-stamp the field-id schema over the same
  buffers). `arrow_cast::cast` allocates even for same-type casts, so the old
  path copied everything. **Measured (100k-row batch):** match **151 ns**
  (660 Gelem/s) vs needs-cast **32 µs** → **~210× faster** on the recast step,
  every write where the batch already matches the table schema (the common case).
- **Gatling multi-core ingest** (`ingest_parallel`) — all-core Parquet encode,
  single ordered writer. **Measured (32×20k rows):** 1 core 123 ms → all cores
  **32 ms** = **~3.8×** (`ingest_parallel_scaling` bench).
- **Read scan:** 10k 15M/s · 100k 14M/s · 1M **19M rows/s** (`read_scan` bench).

## Levers left (prioritised)

1. **Compression knob (zstd/snappy)** — skade writes **UNCOMPRESSED** Parquet
   today, so reads pay full I/O. A per-`Table` compression option (additive)
   trades CPU for I/O; `ingest_parallel` already fans the encode across cores so
   it's parallelised for free. *Next.*
2. **Gatling *no-barrier* ingest** — `ingest_parallel` currently encodes **all**
   files then writes (materialise-all). The znippy gatling pipeline (bounded
   channel: encoders run ahead of one writer) overlaps encode+write and bounds
   memory. Lift `bench/src/pipeline.rs::run_single_writer_ingest`'s channel form.
3. **Projection + predicate pushdown on read** — `read_columns(&[..])` /
   `read_where(expr)` so a scan materialises only the columns/rows asked for
   (the columnar zero-copy win on the read side). `read_all` stays for "give me
   everything".
4. **Writer tuning** — row-group / page sizing, dictionary, `WriterProperties`,
   bigger commits to amortise fsync.
5. **nordisk zero-copy/parallel codec** — the deepest win: `zstd-sys-rs`
   (zero-copy zstd) + `lzip-parallel` as the Parquet codec. Needs a **parquet
   fork** (codec not pluggable) — see ROADMAP item 3.

## Benchmark matrix (`benches/throughput.rs`)

| group | situations |
|---|---|
| `append` | single-batch append, 1k / 10k rows (commit/fsync-bound) |
| `append_partitioned_repo` | per-commit identity partition-key overhead |
| `ingest` | bulk, commit every N (amortised fsync) |
| `ingest_parallel_scaling` | 1-core vs all-cores encode (the gatling payoff) |
| `recast` | match (zero-copy) vs needs-cast — the recast win |
| `read` / `read_scan` | full-snapshot scan, 100k + 10k/100k/1M sizes |

**To add:** compression (none/snappy/zstd) × append/ingest; wide vs narrow
schema; cold-reopen vs warm read; partitioned-prune read once `read_where` lands.

## Container benchmark plan — skade vs Spark+Iceberg

Benchmarks may fire podman containers (same pattern as skade-katalog's
`bench/src/containers.rs` for Nessie/Polaris/RustFS, and knut/znippy). Plan: a
`bench-containers spark` target brings up **Spark + Iceberg** (a
`tabulario/spark-iceberg`-style image) over a shared warehouse, then we
write+read the **same** dataset (e.g. a TPC-H table) through skade and through
Spark and compare write throughput, file sizes, and scan latency — apples to
apples on one box. This is the external baseline; the criterion matrix above is
the internal one.
