# Query engine

The **query engine** (`src/query/`) turns a validated JSON [`QueryDocument`](../src/query/types/document.rs) into a [`QueryResponse`](../src/query/types/response.rs): catalog resolution, a chunk-level **read plan**, and optionally mmap-backed decode plus **`operation`** aggregates.

Flat JSON parsing lives in **`document_wire.rs`**; size/depth limits and **`validate_query`** live in **`document.rs`**. Wire types live in **`types/`** (`document`, `plan`, `response`, `error`). Planning and decode live in **`plan/`** and **`decode/`**; materialize, fold, and dtype routing in **`materialize/`**, **`fold/`**, and **`dispatch.rs`**; orchestration in **`engine/`** (`run`, `operations`, `budget`, `spill_policy`). The engine assumes a parsed, validated document and a mmap’d `.tet` byte slice when planning against a file.

See [JSON security (input and output)](#json-security-input-and-output) for trust boundaries and hardening notes.

## End-to-end flow

```mermaid
flowchart TD
    subgraph control["Control plane outside engine"]
        JSON["Query JSON"]
        Parse["parse_query_json"]
        Validate["validate_query"]
        JSON --> Parse --> Validate
    end

    subgraph entry["engine run.rs"]
        PQ["plan_query_empty"]
        PQM["plan_query_with_tet_mmap"]
        Validate --> PQ
        Validate --> PQM
    end

    subgraph catalog_mmap["catalog mmap"]
        Summary["read_tet_summary_v1"]
        Match{"dataset in catalog?"}
        Chunks["chunk_coords_intersecting_strided"]
    end

    subgraph planning["plan/"]
        Sel["selection resolved_dense_global_box"]
        RP["read_plan build_read_plan"]
        PQM --> Summary --> Match
        Match -->|no| Unmatched["DatasetResolution unmatched"]
        Match -->|yes| Sel --> Chunks --> RP
    end

    subgraph exec["Optional execution"]
        Gate{"preview limit set?"}
        BEP["build_execution_preview"]
        RP --> Gate
        Gate -->|no| RespPlan["QueryResponse read_plan only"]
        Gate -->|yes| BEP
    end

    subgraph out["Response"]
        RespFull["QueryResponse full"]
        Unmatched --> RespFull
        RespPlan --> RespFull
        BEP --> RespFull
        PQ --> Echo["QueryResponse plan-only echo"]
    end
```

**CLI mapping:** `tet query [QUERY]` calls `validate_query` then `plan_query_empty` or `plan_query_with_tet_mmap_ex`. **`-t PATH`** supplies the mmap; **`-x` / `--execute`** runs `build_execution_preview` and sets `raw_f32_preview_max` from **`--preview`** (alias **`--preview-f32`**): default **64** for **`--format full|json`**, **0** for **`stats|quiet`** when omitted; explicit **`0`** with an **`operation`** skips preview arrays but still aggregates; explicit **`0`** with top-level **`"spill"`** (no reduction key) still runs **`mmap_spill`** export.

Success stdout is formatted by [`format_query_response`](../src/query/cli/output/mod.rs) (presentation only — the in-memory [`QueryResponse`](../src/query/types/response.rs) is unchanged for embedders).

| `--format`       | Short | stdout                                                                                                                                         |
| ---------------- | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `full` (default) | —     | Pretty JSON, full `QueryResponse`                                                                                                              |
| `json`           | —     | Compact one-line JSON, full envelope                                                                                                           |
| `stats`          | —     | Slim JSON: status, catalog/read_plan summary, `execution` aggregates — no `read_plan.chunks[]`, no preview arrays                              |
| `plan`           | —     | Slim JSON: catalog + `read_plan` summary only (no `chunks[]`, no `execution`; **`-x`** still runs decode — use without **`-x`** for plan-only) |
| `quiet`          | `-q`  | One line: `dataset=… status=… op=…` + primary aggregate (best after **`-x`** with `operation`)                                                 |

Errors always go to **stderr** with non-zero exit. See [CLI query output](#cli-query-output).

## Module map

| Submodule          | Files                                                                                                                         | Responsibility                                                                                                                                                                  |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`plan/`**        | `selection.rs`, `read_plan.rs`                                                                                                | JSON `selection` → global box + step; `ReadPlan` chunk I/O rows and logical geometry.                                                                                           |
| **`decode/`**      | `chunk_decode.rs`, `indexing.rs`                                                                                              | Mmap slice bounds, codec decode (raw **0**, zstd **1**), scatter; row-major index ↔ coords.                                                                                     |
| **`materialize/`** | `mod.rs`, `parallel.rs`, `int.rs`, `stats.rs`                                                                                 | Full/capped materialize (f32/f64/i32/i64), export spill, parallel fill, tier-C stats.                                                                                           |
| **`fold/`**        | `fold_policy.rs`, `linear_scan.rs`, `shared.rs`, `reduction.rs`, `variance_simd.rs`, `parallel_fold.rs`, `partial_fold.rs`, … | `FoldIoPolicy` / I/O regime; out-of-core **linear scan**; `FoldPlanOutcome`, `ReductionKind`; SIMD bulk **`f32`** sum/sumsq + min/max; parallel + partial-axis streaming folds. |
| **`dispatch.rs`**  | —                                                                                                                             | Dtype routing for materialize, spill, scalar/partial fold.                                                                                                                      |
| **`engine/`**      | `run.rs`, `operations.rs`, `budget.rs`, …                                                                                     | `plan_query_*`, `build_execution_preview`, budget and spill policy.                                                                                                             |
| **`cli/`**         | `history.rs`, `info.rs`, `output/` (`mod.rs`, `plan.rs`, `quiet.rs`, `stats.rs`, …)                                           | Platform query history JSONL; `tet info` formatters; `QueryOutputFormat`, `format_query_response`.                                                                              |

Public re-exports are wired in [`engine/mod.rs`](../src/query/engine/mod.rs) and [`query/mod.rs`](../src/query/mod.rs) (`tetration::query::plan_query_empty`, `format_query_response`, `QueryOutputFormat`, `materialize_read_plan_f32_le`, `ExecutionBudget`, `spill_read_plan_f32_le`, …). Crate root exposes modules plus [`prelude`](../src/lib.rs).

## CLI query output

Library entry: **`format_query_response(&QueryResponse, QueryOutputFormat)`** — builds strings for `tet` stdout without mutating the response.

- **`full` / `json`** — serialize the full envelope (pretty vs compact). Use for debugging, golden tests, and tools that need `read_plan.chunks[]`.
- **`stats`** — [`stats.rs`](../src/query/cli/output/stats.rs) projects catalog/read*plan summaries and execution I/O + `operation*\*` fields; omits chunk rows and preview arrays.
- **`quiet`** — [`quiet.rs`](../src/query/cli/output/quiet.rs) prints a single `key=value` line; partial-axis reductions include reduced shape + sample values. Plan-only and unmatched-dataset cases still emit a short line.

Implement new presentation in **`src/query/cli/output/`**; keep planning/execution in **`engine/`**.

## Planning detail

From `QueryDocument` + catalog metadata:

1. **`plan/selection.rs`** — resolve per-axis box → `g0`, `g1_exclusive`, `step`.
2. **Chunk-touch policy** — if any `step ≠ 1`, use `strided_half_open`; else `dense_half_open_unit_step`.
3. **`catalog`** — `chunk_coords_intersecting_strided` → chunk coord list.
4. **`plan/read_plan.rs`** — `build_read_plan` → `ReadPlan` (chunk I/O rows + `logical_selection_shape`).

Each `ReadPlan.chunks` entry names one on-disk tile that intersects the selection. Chunk iteration order follows the catalog writer (last axis fastest); **decoded values** are **not** in chunk order—they are scattered into logical row-major selection order during materialization.

## Memory budget and execution strategies

Before decode, **`ExecutionBudget::resolve`** picks how much anonymous RAM the engine may use for dense in-memory paths. Precedence (highest wins):

1. Query JSON **`execution.memory_budget_bytes`**
2. Per-file **TIDX header** `memory_budget_bytes` ([`FileExecutionSettingsV1`](../src/catalog/execution.rs); see [`layout_v1.md`](layout_v1.md#chunk-index-header-32-bytes))
3. Query JSON **`execution.memory_budget_percent_bps`** (basis points; **10000 = 100%**, e.g. **2500 = 25%**)
4. Per-file TIDX **`memory_budget_percent_bps`** (**0** = engine default **25%**)
5. **256 MiB** fallback when host RAM cannot be probed

Host RAM is read best-effort via **`utils::host_memory`** (Linux `MemAvailable`, macOS free+inactive pages). Resolved budget and probe results appear on **`execution.memory_budget_bytes`**, **`execution.host_available_ram_bytes`**, **`execution.logical_selection_bytes`**, and (for **`f32`** datasets) **`execution.logical_selection_f32_bytes`** in the response.

**`build_execution_preview`** then picks a **`MemoryStrategy`** (exposed as **`execution.memory_strategy`**):

| Strategy                     | When chosen                                                        | Behavior                                                                                                                                                                                 |
| ---------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`streaming_fold`**         | Tier-A/B **`operation`** (sum, mean, min, …)                       | Single-pass fold (scalar or partial axes); no full logical `Vec`. I/O path is chosen by [`FoldIoPolicy`](../src/query/fold/fold_policy.rs) — see [Adaptive fold I/O](#adaptive-fold-io). |
| **`in_memory_materialize`**  | Tier-C **`operation`** when logical size ≤ budget                  | Full logical decode into RAM; op runs on buffer; preview from same buffer.                                                                                                               |
| **`temp_spill_materialize`** | Tier-C **`operation`** when logical size > budget                  | Full decode to engine temp file under cache allowlist; op via mmap; file deleted when execution finishes.                                                                                |
| **`mmap_spill`**             | Top-level **`"spill": "path"`** and no reduction key               | Full logical selection written row-major **dtype-native LE** to caller path; preview read from spilled file (one decode).                                                                |
| **`capped_in_memory`**       | Preview-only (no `operation`, no spill) when logical size ≤ budget | Capped materialize into **`execution.f32_preview`** or **`execution.f64_preview`** only.                                                                                                 |

When logical selection exceeds the budget and neither **`operation`** nor spill is requested, execution fails with a validation error suggesting a higher budget, an **`operation`**, or spill output.

Capped preview allocates only `min(cap, logical)` elements in the dtype-appropriate preview field (not the full logical tensor). Full materialize (`max_elements: None`) still requires a buffer sized to the logical selection.

### Spill path allowlist

Spill targets come from query JSON (`"spill": "…"`). **Relative paths** resolve against the **`.tet` file’s directory** (not shell cwd).

Default allowed roots (via [`SpillPathAllowlist::default_for_tet`](../src/query/engine/spill_policy.rs), applied automatically with `--tet` + `--execute`):

1. **Canonical `.tet` parent directory** — any resolved spill path **under that directory tree** is allowed (e.g. `data.tet` in `/tmp` → `/tmp/other/out.bin` is valid; use a subdirectory like `/tmp/tet_home/data.tet` if you need a narrower root).
2. **Platform cache/scratch** (best-effort, each as `…/tetration/`):
   - `$XDG_CACHE_HOME/tetration` or `~/.cache/tetration` (Linux and other Unix)
   - `~/.local/cache/tetration` (macOS default when `XDG_CACHE_HOME` unset; also on Linux)
   - Windows: `%LOCALAPPDATA%\\tetration`
   - `$TMPDIR` / `$TEMP` / `$TMP` / `tetration` subdirs when creatable

**`--spill-allow DIR`** (repeatable) **adds** extra roots to that default set.

Example relative spill beside the dataset: `"spill": "slice.bin"` → next to `data.tet`. Example cache spill: `"spill": "/home/you/.cache/tetration/job-42.bin"` (or any path under an allowed root).

### Adaptive fold I/O

Before tier-A/B fold runs, [`FoldIoPolicy::resolve`](../src/query/fold/fold_policy.rs) compares **logical selection bytes** to **~85% of probed host available RAM** (`IN_CORE_IO_HEADROOM_BPS` = 8500). The result drives chunk visit order and whether fold uses **parallel chunks**, **sequential chunks**, or **linear scan**.

```mermaid
flowchart TD
    Resolve["FoldIoPolicy::resolve"]
    Resolve --> Regime{"logical bytes > 85% available RAM?"}
    Regime -->|no| InCore["io_regime: in_core"]
    Regime -->|yes| OutCore["io_regime: out_of_core"]
    InCore --> Par{"chunk_count > 1?"}
    Par -->|yes| Rayon["parallel chunk fold (Rayon)"]
    Par -->|no| Seq1["sequential chunk fold"]
    OutCore --> Dense{"full dense raw scan + contiguous payloads?"}
    Dense -->|yes| Linear["linear_scan: 64 MiB byte stream"]
    Dense -->|no| Seq2["sequential or parallel chunks"]
    Linear --> Read{"CLI -t path set?"}
    Read -->|yes| FileRead["seek + read from file"]
    Read -->|no| MmapSpan["mmap contiguous span"]
```

| Mode                      | When                                                                                                      | I/O pattern                                                                                                     |
| ------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Parallel chunk fold**   | In-core, `chunk_count > 1`, not linear scan                                                               | Rayon over chunks; mmap/decode per chunk; bulk SIMD per slab when geometry allows                               |
| **Sequential chunk fold** | In-core single chunk, or out-of-core without linear-scan eligibility, or `execution.fold_parallel: false` | Chunks in catalog order or ascending `payload_offset` when `sequential_io`                                      |
| **Linear scan**           | Out-of-core + full dense unit-step selection + raw codec + adjacent payloads summing to logical size      | One contiguous byte span; 64 MiB windows; [`push_f32_le_bytes`](../src/query/fold/reduction.rs) SIMD per window |

**Linear scan eligibility:** [`detect_contiguous_raw_span`](../src/query/fold/linear_scan.rs) requires every touched chunk to use **raw codec 0**, payloads **abutting in plan order**, and total raw bytes = logical selection size. Converted multi-chunk **`f32`** grids from `write_raw_array_file` typically satisfy this; zstd chunks or strided partial selections do not.

**Query hint:** `"execution": { "fold_parallel": false }` forces **sequential chunk visits** (offset order when full dense); it does **not** enable linear scan. `"fold_parallel": true` on an out-of-core linear-scan query keeps parallel chunk fold instead of linear scan.

**Execution stats** (also in **`--format stats`**): `io_regime` (`in_core` / `out_of_core` / `unknown`), `fold_parallel`, `fold_linear_scan`.

### Streaming fold performance

**Behavior:** [`streaming_fold`](../src/query/engine/budget.rs) tier-A/B ops route through [`scalar_fold`](../src/query/dispatch.rs). When [`use_parallel_fold`](../src/query/fold/parallel_fold.rs) is true, fold uses Rayon over disjoint chunks (partial [`ValueAccum`](../src/query/fold/reduction.rs) + `merge_from`). When **`fold_linear_scan`** is true, fold uses [`fold_read_plan_scalar_linear`](../src/query/fold/linear_scan.rs) instead — single-threaded byte stream, no per-chunk mmap fan-out.

So a full-file scalar op still **reads every payload byte once** and **touches every element** for the accumulator. Wall time is dominated by **page-cache / disk bandwidth** and CPU over the selection when out-of-core, or **memory bandwidth + cores** when in-core.

**SIMD bulk fold (tier-A/B, `f32` slabs):** [`variance_simd.rs`](../src/query/fold/variance_simd.rs) — NEON on aarch64, SSE2 on x86_64, scalar fallback:

- **mean / sum / var / std** — single-pass sum + sum-of-squares per slab → Welford merge
- **min / max** — single-pass vector min/max per slab → accumulator merge

Used in both parallel chunk fold (dense 1D tile bulk path) and linear scan windows.

**Reference (local, gitignored fixture):** `fixtures/extra_large/h5/tensor_20gb.tet` — 20 GiB logical **`f32`**, 320 chunks × 64 MiB raw (`dataset`: **`data`**). After convert from `tensor_20gb.h5`:

```bash
echo '{"dataset":"data","layout_version":1,"mean":[]}' \
  | tet query -t fixtures/extra_large/h5/tensor_20gb.tet -x -q
```

**In-core** (data fits ~85% of available RAM — e.g. Mac Studio with ~25 GiB free): expect on the order of **~0.5–0.6 s** for full-tensor **mean** and tier-A/B ops over 20 GiB (~30–40 GiB/s effective from warm page cache). Response checks:

- **`execution.memory_strategy`**: `"streaming_fold"`
- **`execution.io_regime`**: `"in_core"`
- **`execution.fold_parallel`**: `true`
- **`execution.fold_linear_scan`**: `false`
- **`read_plan.chunk_count`**: `320`

**Out-of-core** (logical size exceeds headroom — e.g. 20 GiB on a 16 GiB machine with ~6 GiB free): linear scan when payloads are contiguous. Warm 2nd pass on Apple Silicon (May 2026): **~3.7–4.0 s** for mean/sum/std/var/min/max (~2× warm HDF5 on the same fixture). Response checks:

- **`execution.io_regime`**: `"out_of_core"`
- **`execution.fold_linear_scan`**: `true`
- **`execution.fold_parallel`**: `false`

See [`fixtures/bench_results/latest.md`](../fixtures/bench_results/latest.md) for archived runs (`mise run bench:h5`).

A **single-chunk** slice (`selection` stopping after 16 777 216 elements) completes quickly — same fold path, one chunk.

**Parallel vs linear:** per-chunk parallel fold wins when the working set fits page cache. Linear scan wins when parallel mmap/page faults thrash on cold or oversubscribed RAM but payloads are one contiguous raw hyperslab on disk.

## Materialization and operations

```mermaid
flowchart TD
    BEP["build_execution_preview"]
    Budget["ExecutionBudget::resolve"]
    BEP --> Budget
    Budget --> OpGate{"operation set?"}

    OpGate -->|yes| TierGate{"materialize tier?"}
    TierGate -->|streaming| FoldPath["streaming_fold"]
    FoldPath --> Axes{"axes empty?"}
    Axes -->|yes| ScalarFold["fold_read_plan_scalar_operation"]
    Axes -->|no| PartialFold["fold_read_plan_partial_operation"]
    ScalarFold --> OpFields["operation_*"]
    PartialFold --> Reduced["operation_reduced_*"]
    OpFields --> Done["QueryExecutionPreview"]
    Reduced --> Done

    TierGate -->|materialize-required| MatGate{"logical ≤ budget?"}
    MatGate -->|yes| InRam["in_memory_materialize"]
    MatGate -->|no| TempSpill["temp_spill_materialize → delete on drop"]
    InRam --> TierCOp["median, quantile, histogram"]
    TempSpill --> TierCOp
    TierCOp --> Done

    OpGate -->|no| SpillGate{"spill set?"}
    SpillGate -->|yes| Spill["spill_read_plan_*_le → mmap_spill"]
    Spill --> Done

    SpillGate -->|no| CapMat["capped_in_memory materialize"]
    CapMat --> Preview["f32_preview / f64_preview"]
    Preview --> Done
```

- **`materialize_read_plan_f32_le`** / **`materialize_read_plan_f64_le`** — full logical tensor (caller must size for `logical_f32_element_count` and element width).
- **`materialize_read_plan_f32_le_into`** / **`materialize_read_plan_f64_le_into`** — same decode path into a caller-owned buffer (no `Vec` allocation for the output).
- **`materialize_read_plan_f32_le_parallel`** / **`materialize_read_plan_f64_le_parallel`** and **`_into_parallel`** twins — Rayon over planned chunks (raw and zstd). **`build_execution_preview`** uses parallel decode when the read plan touches more than one chunk and materialization is required.
- **`planned_chunk_mmap_slices`** — zero-copy raw codec slices only (no zstd).
- **`spill_read_plan_f32_le`** / **`spill_read_plan_f64_le`** — full logical decode to a caller-owned file path (used by **`mmap_spill`** strategy).

## `QueryResponse` fields (engine-produced)

| Field                                          | When set                                                                                               |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `catalog`                                      | Always with `--tet` / `plan_query_with_tet_mmap`.                                                      |
| `catalog.file_execution`                       | Matched dataset; mirrors TIDX header execution settings.                                               |
| `catalog.dataset_f32_bytes`                    | Matched **`f32`** dataset; `4 × product(shape)`.                                                       |
| `catalog.dataset_f64_bytes`                    | Matched **`f64`** dataset; `8 × product(shape)`.                                                       |
| `read_plan`                                    | Dataset matched; lists touched chunks and selection geometry.                                          |
| `execution`                                    | `raw_f32_preview_max` is `Some(n)` (including `n = 0` when an **`operation`** or **`spill`** is set).  |
| `execution.f32_preview`                        | First `n` logical row-major floats (`n = 0` → empty vec); **`f32`** datasets only.                     |
| `execution.f64_preview`                        | First `n` logical row-major doubles (`n = 0` → empty vec); **`f64`** datasets only.                    |
| `execution.operation_*`                        | Scalar aggregates over full logical selection; preview cap does not truncate them.                     |
| `execution.operation_reduced_*`                | Partial-axis reductions + `operation_reduced_shape`.                                                   |
| `execution.memory_strategy`                    | `streaming_fold`, `capped_in_memory`, `mmap_spill`, `in_memory_materialize`, `temp_spill_materialize`. |
| `execution.io_regime`                          | `in_core`, `out_of_core`, or `unknown` (streaming fold only).                                          |
| `execution.fold_parallel`                      | Whether parallel chunk fold was selected (`streaming_fold`).                                           |
| `execution.fold_linear_scan`                   | Whether out-of-core linear byte-stream fold ran (`streaming_fold`).                                    |
| `execution.memory_budget_bytes`                | Resolved RAM budget used for this run.                                                                 |
| `execution.host_available_ram_bytes`           | Host probe at budget resolution, when available.                                                       |
| `execution.logical_selection_f32_bytes`        | `4 × logical_f32_element_count` (f32 datasets).                                                        |
| `execution.logical_selection_bytes`            | `elem_size × logical_f32_element_count` for the dataset dtype.                                         |
| `execution.spill_f32_path` / `spill_f32_bytes` | After **`mmap_spill`** decode.                                                                         |

## Chunk-touch policy strings

Stable tokens on `ReadPlan.chunk_touch_policy` (see [`CHUNK_TOUCH_POLICY`](../src/query/types/plan.rs)):

- **`dense_half_open_unit_step`** — JSON `step` omitted or 1; chunk list follows dense half-open intervals.
- **`strided_half_open`** — per-axis JSON `step` affects which chunks are touched.

## Related docs

- On-disk layout (TIDX execution header): [`layout_v1.md`](layout_v1.md)
- Axis metadata (dimension names vs coordinates): [`layout_v1.md` — axis metadata](layout_v1.md#axis-metadata-planned-phase-7)
- Roadmap checklist: [`GETTING_STARTED.md`](../GETTING_STARTED.md)

## Dimension names vs coordinate labels (planned)

v1 queries use **decimal axis indices** only (`"mean": 0`). Each **query document** names **one dataset** in the `.tet` file (`"dataset": "temperature"`); multi-dataset join/group is out of scope (see [Tier 3](#tier-3--not-operation-enum-variants)).

Future metadata ([`layout_v1.md — axis metadata`](layout_v1.md#axis-metadata-planned-phase-7)) adds two layers:

### Dimension names (axis names)

- **One string per axis** — the name of the dimension, not each slot along it.
- **Query use (Phase 8):** resolve `"mean": "time"` → internal axis `0` before planning; same reductions as today.
- **Size:** O(`ndim`) — fits in dataset attrs or file header.

### Coordinate labels (index values)

- **One value per position** along an axis — e.g. every row’s timestamp, every column’s variable name, every station id.
- **Query use (later):** slice or filter by label (`time >= "2024-01"`), verify alignment between two datasets on shared coords; **group-by** needs labels **plus** a new aggregation op (not v1).
- **Size:** O(`shape[d]`) — may be stored inline, as attrs, or as a separate 1D array/dataset in the file.

### What labels do _not_ imply

| Capability                      | Needs                                                      |
| ------------------------------- | ---------------------------------------------------------- |
| `"mean": "time"` (axis name)    | **Dimension names** only                                   |
| Slice rows 100–200 by index     | **Today** — numeric `selection`                            |
| Slice by timestamp / station id | **Coordinate labels** + resolver                           |
| Fast lookup without full scan   | Coords **+ optional index** (sorted array, hash, …)        |
| `GROUP BY station`              | Coords on key axis **+ group-by op**                       |
| SQL-style join two datasets     | Two queries or spill + caller; **non-goal** as full engine |

**Append vs join:** growing one dataset over time is a **writer** concern (Phase 7 session writer). Combining two datasets on shared coordinates is **join** semantics — metadata helps **alignment checks**, not automatic join in the query engine.

## Query document (JSON)

The wire format is **flat**: one top-level reduction key per document (`mean`, `sum`, …). Nested `"operation": { … }` is **rejected** at parse time.

| Write                                          | Meaning (internal `operation.axes`)                                       |
| ---------------------------------------------- | ------------------------------------------------------------------------- |
| `"mean": []`                                   | Scalar mean over all selected elements                                    |
| `"mean": 0`                                    | Mean over axis **0** (`["0"]`)                                            |
| `"sum": [0, 1]`                                | Partial reduction over axes 0 and 1                                       |
| `"quantile": { "q": 0.95, "axis": 0 }`         | Quantile on axis 0 (or `"axes": [0, 1]` for multi-axis)                   |
| `"histogram": { "bins": 10, "axis": 0 }`       | Histogram on axis 0                                                       |
| `"execution": { "memory_budget_percent": 40 }` | **40%** of host RAM (`memory_budget_percent_bps` = 4000 internally)       |
| `"execution": { "fold_parallel": false }`      | Force sequential chunk visits (not linear scan); default is policy-driven |
| `"spill": "slice.bin"`                         | Full logical tensor export beside the `.tet` parent (allowlist applies)   |

`memory_budget_percent_bps` is still accepted in JSON for embedders; serialization prefers **`memory_budget_percent`**. Nested `"output": { … }` is rejected (use **`spill`**). Axis indices must be **non-negative decimals** (`0`, not `"time"`) — see [Dimension names vs coordinate labels](#dimension-names-vs-coordinate-labels-planned).

At most **one** reduction key per document. Unknown top-level fields are rejected (`deny_unknown_fields` style in [`parse_query_value`](../src/query/document_wire.rs)).

## Operations (shipped in v1)

Reduction keys map to response fields below. **Scalar** = empty axis list (`[]` on the wire); **partial** = one or more axis indices.

| Operation / JSON key | Scalar (`[]`)                                             | Non-empty axes (partial reduction)                                                                                 |
| -------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `sum`                | `operation_sum`                                           | `operation_reduced_sum` + `operation_reduced_shape`                                                                |
| `mean`               | `operation_mean`                                          | `operation_reduced_mean` + `operation_reduced_shape`                                                               |
| `min`                | `operation_min`                                           | `operation_reduced_min` + `operation_reduced_shape`                                                                |
| `max`                | `operation_max`                                           | `operation_reduced_max` + `operation_reduced_shape`                                                                |
| `count`              | `operation_element_count`                                 | `operation_reduced_count` + `operation_reduced_shape`                                                              |
| `var`                | `operation_var`                                           | `operation_reduced_var` + `operation_reduced_shape`                                                                |
| `std`                | `operation_std`                                           | `operation_reduced_std` + `operation_reduced_shape`                                                                |
| `product`            | `operation_product`                                       | `operation_reduced_product` + `operation_reduced_shape`                                                            |
| `norm_l1`            | `operation_norm_l1`                                       | `operation_reduced_norm_l1` + `operation_reduced_shape`                                                            |
| `norm_l2`            | `operation_norm_l2`                                       | `operation_reduced_norm_l2` + `operation_reduced_shape`                                                            |
| `all_finite`         | `operation_all_finite`                                    | `operation_reduced_all_finite` + `operation_reduced_shape`                                                         |
| `any_nan`            | `operation_any_nan`                                       | `operation_reduced_any_nan` + `operation_reduced_shape`                                                            |
| `arg_min`            | `operation_argmin_index`                                  | `operation_reduced_argmin` + `operation_reduced_shape`                                                             |
| `arg_max`            | `operation_argmax_index`                                  | `operation_reduced_argmax` + `operation_reduced_shape`                                                             |
| `median`             | `operation_median`                                        | `operation_reduced_median` + `operation_reduced_shape`                                                             |
| `quantile`           | `operation_quantile` (`q` field)                          | `operation_reduced_quantile` + `operation_reduced_shape`                                                           |
| `histogram`          | `operation_histogram_counts`, `operation_histogram_edges` | `operation_reduced_histogram_counts` (flat `out_len × bins`, row-major; edges omitted) + `operation_reduced_shape` |

Population **`var` / `std`**, `ddof = 0`. **`norm_l2`** is √(sum of squares).

### Dtypes

| Wire tag | Name  | Writer | Query execution                                 |
| -------- | ----- | ------ | ----------------------------------------------- |
| `1`      | `f32` | yes    | yes (preview in `f32_preview`)                  |
| `2`      | `f64` | yes    | yes (preview in `f64_preview`)                  |
| `3`      | `i32` | yes    | yes (aggregates promote to `f64` in fold paths) |
| `4`      | `i64` | yes    | yes (aggregates promote to `f64` in fold paths) |

Element size and budget math: [`ElementDtype`](../src/utils/dtype.rs).

Quantile example (exact selection on sorted values, linear blend between adjacent ranks):

```json
{ "dataset": "a", "quantile": { "q": 0.95 } }
```

Histogram example (equal-width bins from per-cell min/max; scalar returns edges):

```json
{ "dataset": "a", "histogram": { "axis": 0, "bins": 10 } }
```

Example:

```json
{
  "dataset": "temperature",
  "mean": 0,
  "execution": { "memory_budget_percent": 40 }
}
```

Spill example (full logical tensor to disk, no JSON preview floats required):

```json
{
  "dataset": "temperature",
  "spill": "/tmp/temp_slice.bin"
}
```

**Execution paths today:**

- **Preview only** (no `operation`, no spill) — **`capped_in_memory`** when logical size ≤ budget.
- **Streaming ops** (`sum`, `mean`, …) — **`streaming_fold`**: scalar (empty axis list) via **`fold_read_plan_scalar_operation`**; partial axes via **`fold_read_plan_partial_operation`**. Preview is the first `n` logical values in **`f32_preview`** or **`f64_preview`** when `raw_f32_preview_max > 0`.
- **Materialize-required ops** (`median`, `quantile`, `histogram`) — **`in_memory_materialize`** when logical size ≤ budget; **`temp_spill_materialize`** when over budget (engine temp file under cache allowlist, removed after the op). Requires `--tet` (or explicit **`SpillPathAllowlist`**) so temp paths are allowed.
- **Export spill** — **`mmap_spill`** when **`spill`** is set and no reduction key. Preview is read from the spilled file (single full decode; dtype-native bytes).

Supported dtypes: wire tags in [`DATASET_DTYPE_TAG_V1`](../src/catalog/mod.rs) (`f32` = 1, `f64` = 2, `i32` = 3, `i64` = 4). The preview cap does **not** truncate `operation_*` aggregates.

## Operations roadmap (planned)

New ops should declare which **implementation tier** they use. That keeps “huge tensor + one number” fast while harder stats stay explicit about memory.

### Implementation tiers

| Tier  | Name                             | When to use                                                | Engine pattern                                                                                                                         |
| ----- | -------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **A** | **Scalar fold**                  | `axes: []`, associative or online stats                    | [`scalar_fold`](../src/query/dispatch.rs) → parallel chunks (in-core), linear scan (out-of-core contiguous raw), or sequential chunks. |
| **B** | **Partial-axis fold**            | Non-empty `axes`, element-wise combine along dimensions    | [`partial_fold`](../src/query/fold/partial_fold.rs) with same `FoldIoPolicy` chunk ordering.                                           |
| **C** | **Materialize-required**         | Needs full logical tensor order, sort, or index of extrema | Full decode (or spill file); may add new `operation_*` response fields.                                                                |
| **D** | **Out of scope for `Operation`** | Writers, dtype views, foreign format import                | Separate APIs (`materialize_*`, `tet convert`, metadata), not the JSON `operation` enum.                                               |

### Tier 1 — shipped (v1)

**Done:** tier-A/B streaming ops (`sum`, `mean`, `min`, `max`, `count`, `var`, `std`, `product`, `norm_l1`, `norm_l2`, `all_finite`, `any_nan`, `arg_min`, `arg_max`) — scalar + partial axes.

### Tier 2 — tensor stats (shipped)

**Shipped (index ops):** `arg_min`, `arg_max` (scalar logical index; partial = index within reduced axes fiber).

| Op                           | Tier (typical) | Notes                                                                    |
| ---------------------------- | -------------- | ------------------------------------------------------------------------ |
| **`argmin` / `argmax`**      | C → **A/B**    | **Done** as `arg_min` / `arg_max`.                                       |
| **`median`**                 | C              | **Done** (scalar + partial axes).                                        |
| **`quantile` / `histogram`** | C              | **Done** (scalar + partial axes; histogram partial returns counts only). |

### Tier 3 — not `Operation` enum variants

These match the product vision but belong **beside** the reduction enum:

| Capability                                | Why separate                                                                                                                                        |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Read / export**                         | Plan + materialize or `output.spill` ([`OutputHint::SpillArray`](../src/query/types/document.rs)).                                                  |
| **`cast` / integer dtypes**               | **`i32` / `i64`** on disk and in materialize/query fold (**done**).                                                                                 |
| **Named axis labels**                     | **Dimension names** in metadata → resolve `"time"` to axis index before reductions (Phase 8).                                                       |
| **Coordinate labels / filter-by-value**   | Per-index coords in metadata; slice resolver; optional lookup index (Phase 7 storage + Phase 8+ query).                                             |
| **`rechunk` / resample**                  | Writer / transform path, not read-time aggregate.                                                                                                   |
| **Linear algebra** (`matmul`, `einsum`)   | Belongs in caller libraries on materialized slabs.                                                                                                  |
| **Spectral / ML** (FFT, CWT, conv, train) | Same: materialize or spill, then NumPy / SciPy / PyTorch / JAX — not chunk-local in the engine.                                                     |
| **SQL / joins**                           | Explicit non-goal (see [README](../README.md)).                                                                                                     |
| **CLI query history**                     | **Done** — platform cache JSONL (`tet qhist list` / `run`; `hist` alias); `TET_QUERY_HISTORY_MAX` rotation; `.tet` footer via `tet info --history`. |

### Non-goals for the JSON `operation` field

- Arbitrary per-chunk user callbacks (needs a sandbox and stable ABI).
- Plugin codecs or filters beyond the v1 catalog codec tags.
- Guarantees about numerical order beyond logical row-major **preview** order (aggregates are commutative where noted).
- **FFT, CWT, convolution, and ML ops** — export a hyperslab, then run ecosystem libraries (see Phase 10 Python).
- **Query replay / result cache in `.tet`** — use optional client-side memoization (`tet qhist` stores recent query JSON in platform cache only; does not mutate the file or skip decode by default).

When adding an op, update this table, [`Operation`](../src/query/types/document.rs), `validate_query` / `document.rs`, `reduction.rs` / `operations.rs` / `partial_fold.rs`, and (if tier **C**) `materialize_stats.rs`.

## JSON security (input and output)

The JSON query plane is a **declarative control document**, not executable code. There is no query language interpreter, SQL engine, or “run this string from the JSON” path. Security still matters in **both directions**: untrusted input must not drive memory-unsafe or out-of-policy behavior, and untrusted consumers must not treat output JSON as shell/HTML/SQL without encoding.

### Threat model (v1)

| Surface             | Source                                                   | Risk if mishandled                                                                                        |
| ------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| **Query JSON in**   | User, HTTP body, stdin, agent prompt                     | DoS (huge/deep JSON), logic abuse (absurd selection), confused deputy (if a host passes paths from JSON)  |
| **`.tet` mmap**     | Caller-chosen file path (CLI flag, not JSON field today) | Malicious file → bad index spans (mitigated in [catalog robustness](#robustness-catalog-index))           |
| **Query JSON out**  | `QueryResponse` pretty-print                             | Log/UI injection if embedded raw; unsafe `eval` in downstream scripts                                     |
| **Binary payloads** | Chunk bytes on disk                                      | Not inlined in JSON; decoded only through catalog + read plan                                             |
| **Spill path**      | `"spill"` string in query JSON                           | Host path chosen by caller; validated against [`SpillPathAllowlist`](../src/query/engine/spill_policy.rs) |

The **dataset name** and **operation axis labels** in JSON are echoed in responses and errors. Treat them as **untrusted display data** unless your deployment pre-validates them.

### Input protections (today)

Implemented in [`document.rs`](../src/query/document.rs) and planning:

1. **Typed `serde` deserialization** — JSON maps into fixed structs/enums ([`QueryDocument`](../src/query/types/document.rs), [`Operation`](../src/query/types/document.rs)); there is no dynamic “operation name → callback” table fed by arbitrary strings.
2. **Closed `operation` enum** — Only the v1 ops listed above deserialize; unknown tags fail at parse time.
3. **Axis tokens restricted** — `operation.axes` entries must be non-empty **ASCII decimal** strings (e.g. `"0"`), not arbitrary expressions or Unicode axis names ([`validate_operation_axis_token`](../src/query/document.rs)).
4. **Slice sanity** — `step != 0`; when both `start` and `stop` are set, `start < stop`.
5. **Non-empty `dataset`** — Whitespace-only names rejected.
6. **Catalog binding** — After parse, selection rank/shape and axis indices are checked against the mmap’d dataset ([`plan_query_with_tet_mmap`](../src/query/engine/run.rs), [`resolved_dense_global_box`](../src/query/plan/selection.rs)); JSON cannot name chunk file offsets directly.
7. **No `.tet` path in query JSON** — Opening a `.tet` file uses the **host** path (`--tet`, API argument), not a field inside the query document. Spill **output** paths _are_ in JSON today; deployments should allowlist or rewrite them.
8. **Size and shape caps** — [`parse_query_json`](../src/query/document.rs) rejects payloads over [`QueryLimits::DEFAULT.max_json_bytes`](../src/query/document.rs) (1 MiB) and nesting depth over [`QueryLimits::DEFAULT.max_json_depth`](../src/query/document.rs) (64). [`validate_query`](../src/query/document.rs) caps `dataset` length, `selection` rank and `operation.axes` count (≤ [`MAX_NDIM`](../src/catalog/mod.rs)), and per-axis label length via **`QueryLimits::DEFAULT`**.
9. **`deny_unknown_fields`** — [`QueryDocument`](../src/query/types/document.rs) and nested input types reject unexpected JSON keys at parse time.
10. **Fuzz / property tests** — `src/tests/query.rs` proptest: random UTF-8 must not panic in `parse_query_json` / `validate_query`.

**Not enforced yet (deployments should add limits):**

- Canonical JSON / duplicate-key rejection (depends on `serde_json` behavior).
- Rate limiting and authentication on any HTTP wrapper around `tet query`.
- Spill path allowlist / sandbox directory policy.

### Output protections (today)

1. **`serde_json` serialization** — Strings in `QueryResponse` are JSON-escaped when written (`tet query --format full` pretty-print, or `json` / `stats` compact JSON).
2. **Bounded preview arrays** — `execution.f32_preview` / `execution.f64_preview` are capped by `--preview-f32` / `raw_f32_preview_max`; aggregates use full-tensor math but return numeric summaries, not opaque blobs.
3. **Server-generated messages** — Most `message` text is produced by the engine; user strings appear mainly as echoed `dataset` / axis labels and in validation errors.

### Caller responsibilities (either direction)

#### Ingesting query JSON

- Cap input size and parse time; reject documents above policy limits before calling the library.
- Do not build shell commands, SQL, or file paths by string-concatenating raw JSON fields.
- Keep the `.tet` path under caller control (allowlist directories, no user-supplied absolute paths in multi-tenant services unless intended).
- For **`spill`**, validate or rewrite the path before execution (path traversal, writable location, quota).

**Consuming `QueryResponse` JSON**

- Treat the document as **data**, not instructions: never `eval`, `source`, or template-interpolate response JSON into code without a schema.
- When embedding in HTML, email, or terminals, apply normal contextual escaping (JSON ≠ safe HTML).
- For logs, prefer structured JSON logging or strip/control characters in echoed `dataset` names if logs are human-facing.

### Hardening roadmap

| Item                                       | Direction | Notes                                                                                                       |
| ------------------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------- |
| Input size / depth limits                  | In        | **Done** — `QueryLimits::DEFAULT`                                                                           |
| `deny_unknown_fields`                      | In        | **Done** on query input types                                                                               |
| Dataset / axis length caps                 | In        | **Done** in `validate_query` via `QueryLimits`                                                              |
| Spill path allowlist                       | In/out    | **Done** — `SpillPathAllowlist`, `plan_query_with_tet_mmap_ex`, CLI `--spill-allow`                         |
| Response schema version + stability        | Out       | Document breaking changes                                                                                   |
| Fuzz `parse_query_json` / `validate_query` | In        | **Basic** proptest in `src/tests/query.rs`                                                                  |
| Redaction mode for echoed fields           | Out       | Multi-tenant logging                                                                                        |
| Capped preview without full-buffer alloc   | In        | **Done** — bounded scatter buffer when `max_elements < logical`                                             |
| Parallel streaming fold (tier A/B ops)     | In        | **Done** — Rayon over chunks when in-core; see [Adaptive fold I/O](#adaptive-fold-io)                       |
| Out-of-core linear scan                    | In        | **Done** — contiguous raw hyperslab + SIMD windows; [PR #7](https://github.com/thicclatka/tetration/pull/7) |

## Robustness (catalog index)

- Catalog robustness and index property tests: [`src/tests/catalog.rs`](../src/tests/catalog.rs).
- Query planning, materialize, operations, memory budget, f64 path, tier-C stats: [`src/tests/query.rs`](../src/tests/query.rs).
- Fold I/O policy and contiguous-span detection: [`src/tests/fold.rs`](../src/tests/fold.rs).
- SIMD bulk folds vs scalar reference: [`src/tests/variance_simd.rs`](../src/tests/variance_simd.rs).
- Bulk variance vs Welford reference: [`src/tests/reduction.rs`](../src/tests/reduction.rs).
- Payload decode uses [`src/utils/le_pod.rs`](../src/utils/le_pod.rs) (`f32_le` / `f64_le` / …; bytemuck; aligned cast when possible).

## Intentional gaps (v1)

- Direct callers can still use **`materialize_read_plan_f32_le`** / **`_f64_le`** (always sequential) or **`_parallel`** twins; execution picks parallel decode/fold per `FoldIoPolicy` (see [Adaptive fold I/O](#adaptive-fold-io)).
- **`operation.axes`** uses **decimal dimension indices**, not dimension names or coordinate labels ([planned](#dimension-names-vs-coordinate-labels-planned)).
- Spill path must lie under default roots (canonical **`.tet` parent** and descendants + platform `…/tetration` cache dirs) or `--spill-allow`; relative **`"spill"`** paths resolve against the **`.tet` directory** (not shell cwd).
- Integer dtypes (`i32` / `i64`) are supported in writers and query execution (aggregates promote to `f64` for fold paths).
- JSON hardening: byte size, depth, `deny_unknown_fields`, and string/rank caps are implemented via **`QueryLimits`**; spill path policy is enforced via **`SpillPathAllowlist`** (see [JSON security](#json-security-input-and-output)).
