# Roadmap: anamnesis — Tensor Format Transformation for Rust

> *Parse any format, recover any precision.*

**Date:** March 20, 2026 (updated April 12, 2026)
**Status:** Phases 1–4 complete (v0.4.0 published). FP8/GPTQ/AWQ/BnB dequantization + NPZ parsing + PyTorch `.pth` parsing + GGUF parsing & dequantization (12 block-quant kernels, bit-exact against `llama.cpp` reference). **Next:** Phase 4.5 (GGUF completeness: `IQ*`/`TQ*`/`MXFP4` kernels).
**Context:** The Rust ML ecosystem (candle, burn, tch) cannot load quantized models (FP8, GPTQ, AWQ) or NumPy weight archives (NPZ/NPY for SAEs). The only workaround is a Python script. anamnesis fills this gap: a framework-agnostic, pure-Rust crate that parses tensor formats and recovers precision when needed. Used by hf-fetch-model (download + transform pipeline) and candle-mi (MI framework).

---

## Table of Contents

- [1. Landscape](#1-landscape)
  - [1.1 The Problem](#11-the-problem)
  - [1.2 Existing Solutions](#12-existing-solutions)
- [2. Architecture](#2-architecture)
  - [2.1 Parsing as Foundation](#21-parsing-as-foundation)
  - [2.2 Module Structure](#22-module-structure)
  - [2.3 Ecosystem Fit](#23-ecosystem-fit)
- [3. Phased Development Plan](#3-phased-development-plan)
  - [3.0 Git Workflow](#30-git-workflow)
  - [Phase 1: FP8 Dequantization](#phase-1-fp8-dequantization)
  - [Phase 2: Additional Quantization Schemes](#phase-2-additional-quantization-schemes)
  - [Phase 3: NPZ/NPY Parsing](#phase-3-npznpy-parsing)
  - [Phase 3.5: PyTorch `.pth` Parsing](#phase-35-pytorch-pth-parsing)
  - [Phase 4: GGUF Parsing & Dequantization](#phase-4-gguf-parsing--dequantization)
  - [Phase 4 patch: API polish + dogfooding (v0.4.1)](#phase-4-patch-api-polish--dogfooding-v041)
  - [Phase 4.5: GGUF Completeness](#phase-45-gguf-completeness)
  - [Phase 5: Quantization (Lethe)](#phase-5-quantization-lethe)
  - [Phase 6: Format Conversion Matrix](#phase-6-format-conversion-matrix)
  - [Phase 6.5: Benchmarking & Performance Validation](#phase-65-benchmarking--performance-validation)
  - [Phase 7: Python Bindings (PyO3)](#phase-7-python-bindings-pyo3)
  - [Phase 8: Emerging Quantization Formats](#phase-8-emerging-quantization-formats)
  - [Phase 9: CPU SIMD Pass](#phase-9-cpu-simd-pass)
  - [Future Directions](#future-directions)
- [4. Key Design Decisions](#4-key-design-decisions)
- [5. Relationship to Other Projects](#5-relationship-to-other-projects)

---

## 1. Landscape

### 1.1 The Problem

Quantized models are the norm. Mistral ships Ministral 3B as FP8 safetensors. Community quantizers (TheBloke, NeuralMagic, etc.) distribute Llama and other models in GPTQ and AWQ. SAE weights (Gemma Scope) ship as NPZ archives. None of these load in any Rust ML framework:

```rust
let vb = VarBuilder::from_mmaped_safetensors(&[path], DType::BF16, &device)?;
// → Error: unsupported safetensor dtype F8_E4M3
```

This blocks candle, candle-mi, burn, and tch. The entire Rust ML ecosystem stops at the file format boundary.

### 1.2 Existing Solutions

*Surveyed March 2026. No framework-agnostic tensor format transformation library exists in any language — not in Rust, not in Python, not in C++. Dequantization is universally implemented ad-hoc inside inference engines.*

#### Rust ecosystem

| Solution | What it does | Why it's not enough |
|---|---|---|
| [`float8`](https://crates.io/crates/float8) v0.7.0 | FP8 primitive types (E4M3, E5M2) with `to_f32()` / `from_f32()` conversions | Type-level only. No tensors, no safetensors, no scale factors, no block-wise dequant. A useful **building block** — anamnesis should depend on it for FP8 ↔ f32 element conversion. |
| [candle](https://github.com/huggingface/candle) v0.9.2+ | Added `F8E4M3` DType. [DeepSeek V3 PR #2745](https://github.com/huggingface/candle/pull/2745) implements block-wise FP8 dequant with `scale_inv` | Dequant logic is **model-specific** (DeepSeek V3 only), embedded in `candle-transformers`. Not a reusable library. No E5M2. Still fails on generic FP8 safetensors: `unsupported safetensor dtype F8_E4M3`. |
| [mistralrs-quant](https://lib.rs/crates/mistralrs-quant) | GPTQ, AWQ, HQQ, BnB, FP8 via `QuantMethod` trait with `dequantize_w()` | **CUDA-only** for GPTQ/AWQ. Tightly coupled to mistral.rs inference (742K SLoC, depends on candle-core, tokio, rayon). Not reusable as a standalone library. |
| [pmetal-gguf](https://docs.rs/pmetal-gguf) | Standalone GGUF dequantization with SIMD (`dequant::dequantize()`) | **GGUF-specific.** Does not handle safetensors FP8, GPTQ, AWQ, or NPZ. |
| [`gguf-rs-lib`](https://crates.io/crates/gguf-rs-lib) v0.2.5 | Type-safe GGUF reader/writer. Pure safe Rust. | Parsing only — no dequantization. Potential Phase 4 dependency (TBD). |
| [`llama-gguf`](https://crates.io/crates/llama-gguf) v0.14.0 | Pure Rust llama.cpp reimplementation with GPU dequant kernels (CUDA/Metal/DX12/Vulkan). | Full inference engine, not a standalone library. K-quant dequant math is a useful reference. |
| [`npyz`](https://crates.io/crates/npyz) v0.8.4 | Mature NPY/NPZ parser (4.6M downloads). f16 support via `half` feature. Full read/write. | **No bf16.** Initial Phase 3 approach wrapped `npyz`, but benchmarking revealed 23× overhead vs raw I/O. Replaced with anamnesis's own fast parser (17.7× faster). |
| [`ndarray-npy`](https://crates.io/crates/ndarray-npy) v0.10.0 | Most popular NPY/NPZ crate (7.2M downloads). Actively maintained. | **No f16/bf16.** Tightly coupled to ndarray. Less suitable than `npyz` for framework-agnostic use. |
| [safetensors](https://crates.io/crates/safetensors) | Reads/writes safetensors files | No quantization awareness. Pure format I/O. |
| [burn](https://github.com/tracel-ai/burn) v0.14.0+ | INT8 quantization (beta) | No FP8, no GPTQ, no AWQ. Own internal format only. |
| tch-rs | Defines FP8 variants (`Float8e4m3fn`, `Float8e5m2`) | Wraps PyTorch C++ API. Requires full libtorch installation. No Rust-native dequant. |
| candle-mi's NPZ parser | Loads Gemma Scope SAE weights from `.npz` | Tightly coupled to candle. Not reusable outside candle-mi. |

#### Python ecosystem

| Solution | What it does | Why it's not enough |
|---|---|---|
| [compressed-tensors](https://github.com/vllm-project/compressed-tensors) (vLLM) | Safetensors extension for sparse/quantized storage. Closest conceptual competitor to anamnesis. | Python-only, vLLM-coupled, decompression-to-bf16 still immature ([users requesting basic examples](https://huggingface.co/moonshotai/Kimi-K2-Thinking/discussions/2) as of late 2025). |
| [optimum-quanto](https://github.com/huggingface/optimum-quanto) (HuggingFace) | In-memory quantize/dequantize on PyTorch tensors | Being merged into `optimum`. In-memory only — no file-to-file conversion. PyTorch-coupled. |
| [TorchAO](https://github.com/pytorch/ao) | INT4, INT8, FP8 quantization within PyTorch | PyTorch tensors only, not a file format tool. |
| transformers `dequantize=True` | Loads fine-grained FP8 with dequantization during model load | Embedded in model loading pipeline. Not a standalone operation. |
| AutoGPTQ → [GPTQModel](https://pypi.org/project/GPTQModel/) | GPTQ quantization/dequantization | Dequant happens in fused CUDA kernels during inference. No standalone API. AutoGPTQ archived April 2025. |
| AutoAWQ, bitsandbytes | AWQ and NF4/INT8 quantization | Same pattern: dequant is implicit during inference, not exposed as a file conversion. |
| [llm-compressor](https://github.com/vllm-project/llm-compressor) (vLLM) | "Model-free PTQ" on safetensors (FP8) | **Quantization direction only.** No dequantization. |

#### C/C++ ecosystem

| Solution | What it does | Why it's not enough |
|---|---|---|
| llama.cpp | Converts safetensors → GGUF (Python scripts); dequantizes GGUF during inference (C++) | GGUF-specific. Not a library. Conversion scripts are Python. |
| NVIDIA TensorRT | `IQuantizeLayer` / `IDequantizeLayer` in graph IR | Operates within TensorRT's graph representation, not on raw files. NVIDIA-coupled. |
| NVIDIA TransformerEngine | FP8 compute library (matmul kernels) | Compute library, not a format conversion tool. |

#### The gap

No tool in any language provides **file → transform → file** as a first-class, framework-agnostic operation for quantized tensor formats. Dequantization is always buried inside inference engines or model loading pipelines. anamnesis is the first library designed for this purpose.

---

## 2. Architecture

### 2.1 Parsing as Foundation

Every operation in anamnesis begins with parsing. You cannot remember what you have not first parsed. You cannot quantize what you have not first parsed. You cannot inspect what you have not first parsed.

Parsing is the act of making contact with the weights: decoding their structure, validating their format, understanding what is present and what Lethe took. Sometimes parsing alone is enough (NPZ — nothing was forgotten). Other times, parsing reveals that precision recovery is needed (FP8 — Lethe took something), and remembering follows.

### 2.2 Module Structure

```
anamnesis (library crate)
│
├── parse/              ← decode + validate any tensor format
│   ├── safetensors        safetensors (including quantized metadata)
│   ├── npz                NPZ/NPY archives (feature-gated)
│   ├── pth                PyTorch .pth state_dict (feature-gated, Phase 3.5)
│   └── gguf               GGUF (feature-gated, Phase 4)
│
├── remember/           ← built on parse: precision recovery (dequantize)
│   ├── fp8                fine-grained, per-channel, per-tensor FP8 (E4M3, E5M2)
│   ├── gptq               GPTQ dequantization
│   ├── awq                AWQ dequantization
│   ├── bnb                BitsAndBytes (NF4, INT8) dequantization
│   ├── pth                PyTorch .pth → safetensors conversion (Phase 3.5)
│   └── gguf               GGUF K-quant dequantization (Phase 4)
│
├── lethe/              ← built on parse: precision reduction (quantize)
│
├── inspect             ← built on parse: report without transforming
│
└── bin/
    └── main.rs         ← CLI binary (feature-gated behind "cli")
                           installed as both `anamnesis` and `amn`
                           subcommands: parse, inspect/info, remember/dequantize, forget/quantize
```

### 2.3 Ecosystem Fit

```
safetensors / npz / pth / gguf  ← file formats
    ↓
anamnesis (library)     ← parse, remember, lethe
anamnesis / amn (CLI)   ← amn parse, amn inspect, amn remember, amn forget
    ↓
hf-fetch-model          ← download + transform pipeline
                           (hf-fm --dequantize bf16, download_and_parse_npz)
candle / candle-mi      ← load + run + interpret
burn / tch / ...        ← any Rust ML consumer
```

hf-fetch-model depends on anamnesis for two things:
1. `--dequantize bf16` calls `anamnesis::parse()` + `.remember()` after download.
2. `download_and_parse_npz()` calls `download()` + `anamnesis::parse_npz()`.

---

## 3. Phased Development Plan

### 3.0 Git Workflow

Single `main` branch. Each task ends with a commit. Phases end with a push and a version tag. Same workflow as candle-mi and hf-fetch-model.

Commit style: imperative mood, lowercase, no trailing period. Examples:
- `add safetensors header parsing`
- `implement fine-grained FP8 dequantization`
- `validate FP8 dequantization against EXAONE-4.0-1.2B`

### Phase 1: FP8 Dequantization

**Goal:** The flagship use case — parse an FP8 safetensors file, dequantize to BF16, write a standard safetensors file loadable by any Rust ML framework. Proves the parse → remember architecture end-to-end.

- [x] Scaffold crate:
  - `git init` + GitHub repo creation + remote setup
  - `cargo init --lib`
  - `Cargo.toml` with metadata (name, version `0.1.0`, description, license MIT OR Apache-2.0, keywords, categories), feature gates (`cli = ["dep:clap"]`, `npz = ["dep:npyz"]`), two `[[bin]]` targets (`anamnesis` and `amn`, both pointing to `src/bin/main.rs`, `required-features = ["cli"]`) following the hf-fetch-model pattern
  - `src/lib.rs` with `#![forbid(unsafe_code)]`, `#![deny(warnings)]`
  - `CLAUDE.md` — "Before writing or modifying any Rust code, read and follow CONVENTIONS.md." (same as hf-fetch-model)
  - `LICENSE-MIT` + `LICENSE-APACHE` — dual license files
  - `.gitignore` — standard Rust (`/target`, etc.)
  - `.github/workflows/ci.yml` + `.github/workflows/publish.yml` — reused from hf-fetch-model's pattern with `actions/checkout@v5`: format check, clippy (`--all-targets` + `--all-features`), tests, doc check on publish, tag-triggered `cargo publish`
  — **commit**
- [x] Safetensors parsing foundation (`src/parse/safetensors.rs`) — read header JSON, extract tensor names, shapes, dtypes, byte offsets. Identify quantized tensors (FP8) vs passthrough (BF16/F32 norms, embeddings). Detect fine-grained FP8 metadata (scale factor tensors, block structure). This is the "make contact" layer — **commit**
- [x] Inspect (`src/inspect.rs`) — built on parse. Report format, tensor count, quantized vs passthrough breakdown, size estimate (current and dequantized). Matches the flagship CLI output in `amn-flagship-v2.md` — **commit**
- [x] Fine-grained FP8 dequantization (`src/remember/fp8.rs`) — E4M3 with 128×128 block scale factors. Scalar implementation written for auto-vectorization: contiguous `&[u8]` → `&mut [u16]` slices, no branches in the hot path (bitwise select for NaN/subnormal), `chunks_exact(128)` matching block size, scale factor hoisted before inner loop. Verify with `cargo-show-asm` that the compiler emits SIMD instructions; annotate with `// VECTORIZED:`. Built on the parsed representation from `parse/safetensors` — **commit**
- [x] Per-tensor FP8 dequantization — single scale factor per tensor (simpler case). Same module, same SIMD-friendly loop structure, different scale broadcast — **commit**
- [x] Parse-first public API (`src/lib.rs`) — `parse(path)` returns a `ParsedModel` struct holding header metadata + byte data. `ParsedModel::inspect()` returns format info. `ParsedModel::remember(output_path, TargetDtype)` dequantizes and writes a standard safetensors file. No file is re-read after the initial parse. The Rust API in `amn-flagship-v2.md` should work — **commit**
- [x] CLI binary (`src/bin/main.rs`) — thin `clap` wrapper over the library API. Subcommands: `parse`, `inspect` (alias `info`), `remember` (alias `dequantize`). Each subcommand calls `anamnesis::parse()` then the appropriate method. Progress output via `indicatif` (optional, behind `indicatif` feature). Same binary serves both `anamnesis` and `amn` names — **commit**
- [x] Download FP8 test models via `hf-fetch-model` (v0.8.1, with `list-files`). 7 models from 5 quantization tools, covering 3 FP8 schemes discovered during validation:

  | Model | Size | FP8 scheme | Scale dtype | Quantizer |
  |---|---|---|---|---|
  | `LGAI-EXAONE/EXAONE-4.0-1.2B-FP8` | 1.39 GB | **Fine-grained** | BF16 | LG AI |
  | `Qwen/Qwen3-1.7B-FP8` | 2.47 GB | **Fine-grained** | BF16 | Qwen |
  | `Qwen/Qwen3-4B-Instruct-2507-FP8` | 4.83 GB | **Fine-grained** | **F16** | Qwen |
  | `mistralai/Ministral-3-3B-Instruct-2512` | 4.35 GB | **Per-tensor** (scalar) | BF16 | Mistral |
  | `RedHatAI/Llama-3.2-1B-Instruct-FP8` | 1.88 GB | **Per-tensor** | BF16 | RedHat |
  | `RedHatAI/Llama-3.2-1B-Instruct-FP8-dynamic` | 1.89 GB | **Per-channel** `[N,1]` | BF16 | RedHat |
  | `nvidia/Llama-3.1-8B-Instruct-FP8` (shard 1) | 4.65 GB | **Per-tensor** (scalar) | F32 | NVIDIA |

  Discoveries during validation:
  - **Per-channel FP8** — a third scheme (one scale per row, shape `[N,1]`), used by RedHat/vLLM dynamic quantization. New `dequantize_per_channel_fp8_to_bf16()` and `PerChannelFp8` scheme variant.
  - **F16 scales** exist in the wild (Qwen3-4B-Instruct-2507-FP8, Qwen/Alibaba), undocumented in the ecosystem. All three scale dtypes (F32, BF16, F16) now supported.
  - **Scheme detection by scale shape**, not name suffix — both fine-grained and per-tensor use `_scale_inv`, distinguished by scale tensor dimensionality.
  — **commits** (multiple: BF16 scale fix, scheme detection fix, F16 + per-channel support)

- [x] Validation against FP8 test models — cross-validated against PyTorch (`torch.float8_e4m3fn` → `torch.bfloat16`) on 256×256 slices from all 7 models. **Bit-exact match** (0 ULP difference on 65,536 elements per fixture). Fixtures generated by `tests/fixtures/fp8_reference/generate.py`, Rust tests in `tests/cross_validation.rs`. Auto-vectorization verified with `cargo-show-asm`: SSE2 (default), AVX2 (`target-cpu=native`). **2.7–9.7× faster than PyTorch** (AVX2). — **commit** — **PUSH**

**Deliverable:** `anamnesis` v0.1.0 — FP8 dequantization works. — **PUSH + tag `v0.1.0`**

**Dependencies:** `safetensors`, `half`, `float8` (FP8 ↔ f32 element conversion). CLI: `clap` 4 with `derive` (behind `cli` feature), `indicatif` (behind `indicatif` feature, optional progress bars).

**Note on cached test assets:** The local HuggingFace cache (`~/.cache/huggingface/hub/`) contains 21 models, none with FP8/GPTQ/AWQ quantization. The one asset relevant to anamnesis is **google/gemma-scope-2b-pt-res** which contains `params.npz` (302 MB, 5 F32 arrays: `W_dec`, `W_enc`, `b_dec`, `b_enc`, `threshold`) — this will be used for Phase 3 NPZ validation.

**Note on candle-mi auto-config:** Validating EXAONE-4.0 and Qwen3 after dequantization requires extending candle-mi's auto-config with `exaone4` and `qwen3` `model_type` support. These are both text-only decoder transformers close to existing supported families:
- `exaone4` — LLaMA-like with alternating sliding window / full attention layers ("LLLG" pattern). Similar to Gemma 2's alternating window scheme.
- `qwen3` — extends `qwen2` with QK LayerNorm (additional `q_norm` / `k_norm` tensors per layer) and thinking mode support. The core architecture is otherwise identical to `qwen2`.

### Phase 2: Additional Quantization Schemes

**Goal:** Extend `remember/` to cover the major quantization formats. Each scheme adds a new submodule under `remember/` and extends the parsing layer to detect its metadata.

- [x] GPTQ dequantization (`src/remember/gptq.rs`) — INT4/INT8 with group-wise scale + zero-point. Parse GPTQ metadata from safetensors, reconstruct full-precision weights. Bit-exact against PyTorch on 4 real models (2 quantizers × 2 bit widths), 6.5–12.2× faster than CPU PyTorch (AVX2). Loop fission for full AVX2 vectorization — **commit**
- [x] AWQ dequantization (`src/remember/awq.rs`) — activation-aware INT4 with per-group scales, no +1 zero-point offset. Packs along out_features (columns), unlike GPTQ (rows). Bit-exact against PyTorch on 2 real models (AutoAWQ GEMM), 4.7–5.7× faster than CPU PyTorch (AVX2). Loop fission applied from the start. Note: 8-bit AWQ path is unit-tested but no real 8-bit AWQ models exist in the standard AutoAWQ `.qweight` format — all "8-bit AWQ" on HuggingFace use `compressed-tensors` (vLLM) or are mislabeled 4-bit — **commit**
- [x] BitsAndBytes dequantization (`src/remember/bnb.rs`) — NF4, FP4, double-quant NF4/FP4, and INT8 (LLM.int8()). 4-bit uses 16-entry lookup table + per-block absmax; INT8 uses per-row absmax / 127.0. Bit-exact against PyTorch on 4 real models, 18–54× faster for NF4/FP4 (AVX2), 1.2× for INT8 (near bandwidth limit). Loop fission for NF4/FP4; single-pass AVX2 for INT8 (`vpmovsxbd` → `vcvtdq2ps` → `vmulps`). Format discovery via `hf-fm inspect` (remote header probing) — **commit**
- [x] Feature gates — each scheme behind its own feature flag (`gptq`, `awq`, `bnb`). Default features: `fp8` only — **commit**
- [x] Validation against real models for each scheme — **commits as needed** — **PUSH**

**Deliverable:** `anamnesis` v0.2.0 — all major quantization schemes supported. — **PUSH + tag `v0.2.0`**

**New dependencies:** None expected (pure bit manipulation), but feature-gated if any arise.

### Pre-Phase 3: BnB4 output shape recovery

**Goal:** BnB4 weights are stored flattened (`[N, 1]`). The current dequantization emits a 1D `[total_elements]` output shape (see `model.rs`). The original 2D shape (`[out_features, in_features]`) must be recovered from `config.json` for downstream consumers that expect shaped weight tensors.

- [x] Recover original weight shape for `BnB4` tensors from `quant_state.bitsandbytes__nf4`/`__fp4` companion tensor (`JSON` blob with `"shape"` field). No `config.json` needed — the shape is stored inside the safetensors file itself — **commit**

### Phase 3: NPZ/NPY Parsing

**Goal:** Extend the `parse/` layer to NumPy archives. This is the case where parsing alone suffices — nothing was forgotten, the weights just need extracting from a foreign container. Migrates candle-mi's tightly-coupled NPZ parser into anamnesis as a reusable, framework-agnostic module.

**Own the fast path.** Initial implementation wrapped `npyz` (v0.8.4), but benchmarking revealed 23× overhead vs raw I/O due to per-element deserialization. Replaced with a custom `NPY` header parser (~80 lines) and bulk `read_exact` — for LE data on LE machines (>99% of ML files), the raw bytes ARE the correct output. Uses `zip` v2 directly for `ZIP` extraction. Result: 84 ms for 302 MB (1.3× raw I/O overhead), **17.7× faster** than `npyz`, **4.9× faster** than candle-mi's custom parser.

- [x] NPZ integration module (`src/parse/npz.rs`) — custom `NPY` header parser with bulk `read_exact` data extraction. `parse_npz(path) -> HashMap<String, NpzTensor>`. Zero per-element deserialization for LE data on LE machines. BF16 interpretation layer (JAX `V2` void dtype). **3,586 MB/s** on 302 MB file (1.3× raw I/O overhead, 17.7× faster than `npyz`) — **commit**
- [x] `NpzTensor` type — framework-agnostic struct: `name: String`, `shape: Vec<usize>`, `dtype: NpzDtype`, `data: Vec<u8>`. The `NpzDtype` enum includes `BF16` which `NumPy` cannot represent natively — **commit**
- [x] Feature-gated behind `npz` feature (adds `zip` v2 with `deflate`) — **commit**
- [x] Validation against Gemma Scope SAE weights — already cached locally at `~/.cache/huggingface/hub/models--google--gemma-scope-2b-pt-res/` (`params.npz`, 302 MB, 5 F32 arrays: `W_dec` [16384×2304], `W_enc` [2304×16384], `b_dec` [2304], `b_enc` [16384], `threshold` [16384]). Parse, verify shapes and values match Python reference — **commit** — **PUSH**

**Deliverable:** `anamnesis` v0.3.0 — NPZ parsing works. candle-mi can migrate its NPZ dependency from internal to `anamnesis`. — **PUSH + tag `v0.3.0`**

**New dependencies:** `zip` v2 with `deflate` feature (direct dependency, no `npyz`). Feature-gated behind anamnesis's `npz` feature.

### Phase 3.5: PyTorch `.pth` Parsing

**Goal:** Add PyTorch `.pth` (state_dict) parsing and lossless conversion to safetensors. This enables loading pre-trained weights from repositories that only ship `.pth` files (e.g., AlgZoo's 400+ tiny models on GCS, plus thousands of older HuggingFace models). Unlike Phases 1–2, no dequantization is needed — `.pth` tensors are already full-precision (F32/F64/BF16/F16).

**Approach:** Implement a minimal pickle VM (~36 opcodes, not the full protocol) that interprets the `data.pkl` stream inside the ZIP archive, extracts tensor metadata (shape, dtype, storage reference), and reads raw bytes from `data/{index}` entries. Security boundary: explicit GLOBAL allowlist rejects non-`torch.*` callables. Lossless format conversion to safetensors via `safetensors::tensor::serialize_to_file`. Feature-gated behind `pth`.

**Detailed plan:** See `PLAN-PTH-v0.3.1.md`.

- [x] Extract `byteswap_inplace` from `npz.rs` to shared `src/parse/utils.rs` — **commit**
- [x] Pickle VM + tensor extraction (`src/parse/pth.rs`) — minimal stack machine, GLOBAL allowlist, storage cache for shared storage, stride handling — **commit**
- [x] Test fixtures — 3 AlgZoo `.pth` models (2–3.5 KB each) from GCS + Python-generated JSON reference manifests — **commit**
- [x] Cross-validation tests (`tests/cross_validation_pth.rs`) — byte-exact comparison against `PyTorch` on 3 AlgZoo models (RNN + Transformer, newer and older ZIP formats) — **commit**
- [x] Safetensors conversion (`src/remember/pth.rs`) — lossless format writer, byte-exact roundtrip verified — **commit**
- [x] `ParsedPth` + `PthInspectInfo` — inspect/to_safetensors methods, Display impl — **commit**
- [x] CLI integration (`src/bin/main.rs`) — format dispatch by extension + ZIP magic, `parse`/`inspect`/`remember` for `.pth` — **commit**
- [x] Module wiring — `mod.rs`, `lib.rs`, `Cargo.toml` feature gate, `error.rs` cfg update — done incrementally across prior commits

**Deliverable:** `anamnesis` v0.3.1 — PyTorch `.pth` state_dict parsing + safetensors conversion. — **PUSH + tag `v0.3.1`**

**New dependencies:** None (reuses `zip` v2 from `npz`). Feature-gated behind `pth`.

### Phase 4: GGUF Parsing & Dequantization

**Status:** Complete (`v0.4.0` published). Parser + all 12 block-quant dequantisation kernels + streaming API + cross-validation against `llama.cpp` reference (10/12 kernels bit-exact, 6.3–31.3× faster than Python reference).

**Goal:** Add support for GGUF, the dominant format for local inference (~166,000 models on HuggingFace as of March 2026). GGUF is to llama.cpp/Ollama/LM Studio what safetensors is to HuggingFace Transformers. No standalone, framework-agnostic Rust crate exists for GGUF dequantization — candle-core has GGUF support but it is tightly coupled to candle's tensor types and inference pipeline.

**Approach:** Parse GGUF metadata and tensor layout. Dequantize K-quant types (`Q2_K`–`Q8_K`) and legacy types (`Q4_0`, `Q4_1`, `Q5_0`, `Q5_1`, `Q8_0`, `Q8_1`) to BF16 using the same loop-fission + auto-vectorization strategy as GPTQ/AWQ/BnB. Feature-gated behind `gguf`.

**Key projects studied:**
- **ggml-org/ggml** (`src/ggml-common.h`, `src/ggml-quants.c`) — canonical block struct layouts with `_Static_assert` byte counts and scalar `dequantize_row_*` reference implementations. **Used as the ground truth for every kernel in this phase** (ported verbatim with CONVENTIONS annotations applied).
- **candle-core `quantized`** (`k_quants.rs`) — secondary math reference for K-quants; not depended on.
- **gguf-rs-lib** — considered as a parsing dependency but rejected in favour of a lean in-house parser (matches the `.npy` and `.pth` pattern, keeps all error paths inside `AnamnesisError`).
- **llama-gguf (Lexmata)** — pure Rust reimplementation of llama.cpp with GPU dequant kernels; not used.

- [x] **GGUF file parser** (`src/parse/gguf.rs`, commit `2acaf1a`) — lean in-house parser for `GGUF` v2 and v3. Memory-mapped via `memmap2` (reused from `pth`), returns `Cow::Borrowed` tensor views with zero per-tensor heap allocation. Reads header, metadata KV pairs (all 13 value types including nested `ARRAY`), and the tensor info table; resolves absolute tensor-data offsets from the `general.alignment` metadata key (default 32 B). Adversarial-input DoS guards on tensor count (1 M), KV count (1 M), string length (16 MiB), array length (16 M), array nesting depth (4), tensor dimensions (8), and element product (1 T). Public types: `GgufType`, `GgufMetadataValue`, `GgufMetadataArray`, `GgufTensor`, `GgufTensorInfo`, `GgufInspectInfo`, `ParsedGguf`. **21 unit tests.** **commit**
- [x] **K-quant dequantization** (`src/remember/gguf.rs`, commit `cc4ecf8`) — scalar reference kernels for `Q2_K`, `Q3_K`, `Q4_K`, `Q5_K`, `Q6_K`, `Q8_K` (256-element super-blocks). Each block is processed with two-pass loop fission: pass 1 unpacks the packed-bit storage into an `[f32; 256]` stack scratch buffer (L1-resident), pass 2 walks the scratch and emits `BF16` bytes via the shared `f32_bits_to_bf16_bits` helper from `remember::fp8`. `get_scale_min_k4` (Q4_K/Q5_K's 6-bit packed-scale extractor) and `q3_k_unpack_scales` (Q3_K's `kmask1`/`kmask2` permute) are ported verbatim as private `#[inline]` helpers with their own unit tests — the two most error-prone bit-packing pieces are verifiable in isolation. **commit**
- [x] **Legacy quant types** (`src/remember/gguf.rs`, commit `cc4ecf8`) — scalar kernels for `Q4_0`, `Q4_1`, `Q5_0`, `Q5_1`, `Q8_0`, `Q8_1` (32-element blocks). `Q8_1` is a trivial variant of `Q8_0` (same `d * qs[j]` formula; the auxiliary `d × Σ qs` field is ignored for reconstruction). Landed together with the K-quants in a single commit — the loop-fission template is identical; only the per-type pass-1 unpack differs. **commit**
- [x] **Streaming dequant API** (`dequantize_gguf_blocks_to_bf16`, commit `b6610fe`) — `FnMut(&[u8]) -> Result<()>` sink closure receives one block's worth of `BF16` bytes at a time (64 B legacy, 512 B K-quant). Peak heap is O(one scratch + one block buffer) ≈ 1.5 KB, independent of tensor size — enabling dequantisation of 70 B-parameter models on modest-RAM machines by streaming directly to disk. The `Vec`-returning `dequantize_gguf_to_bf16` is now a thin wrapper around the streaming variant; both entry points share the same validation and the same scalar kernels via a generic `run_legacy_kernel` / `run_super_kernel` outer-loop helper.
- [x] **Algorithmic audit fixes + consistency pass** (commits `b6610fe`, `3156104`) — `Vec::with_capacity` + `extend_from_slice` instead of `vec![0u8; n]` (no zero-init memset); `chunks_exact` outer loop eliminates per-block bounds checks; infallible `read_f16_bytes([u8; 2]) -> f32` / `read_f32_bytes([u8; 4]) -> f32` replace the old `Result`-returning readers; `n_elements.checked_mul(2)` overflow guard catches 32-bit targets with > 2 GiB of `BF16` output; `read_scales12` helper deduplicates the 12-byte packed-scale load across Q3_K/Q4_K/Q5_K. **30 unit tests** after the refactor.
- [x] **Feature-gated behind `gguf` feature** — added in commit `2acaf1a` as part of the parser commit (`gguf = ["dep:memmap2"]` in `Cargo.toml`). Reuses `memmap2` (already a `pth` dep) and `half` (already mandatory). No new third-party crate in any Phase 4 commit.
- [x] **Cross-validation against `llama.cpp` reference dequantization** (commit `73f8e74`) — 10 of 12 production kernels bit-exact (0 ULP) against the `gguf` Python package's `dequantize` function (the official `ggml-org` reference mirroring `ggml-quants.c`'s `dequantize_row_*`). Legacy quants: `Q4_0`, `Q4_1`, `Q5_0`, `Q5_1`, `Q8_0`. K-quants: `Q2_K`, `Q3_K`, `Q4_K`, `Q5_K`, `Q6_K`. Fixtures from bartowski SmolLM2-135M-Instruct and TheBloke TinyLlama-1.1B-Chat (65 536-element slices). `Q8_1` and `Q8_K` are not shipped by any real model (internal `llama.cpp` activation quant types) and are already covered by unit tests. **6.3–31.3× faster** than the Python reference (AVX2). **10 integration tests.** — **PUSH + tag `v0.4.0`**

**Deferred follow-ups:** every out-of-scope item is rehomed to **Phase 4.5: GGUF Completeness** (`IQ*`/`TQ*`/`MXFP4` kernels — targeting `v0.4.2`) and **Phase 9: CPU SIMD Pass** (cross-format AVX2/NEON pass-2 SIMD). User-facing polish (GGUF CLI subcommands, `ParsedGguf::dequantize_tensor`) is pulled into v0.4.1 alongside the in-memory safetensors API from the candle-mi dogfooding report. Big-endian GGUF v3 support has no committed target — the parser detects byte-swapped magic and returns a clear `Unsupported` error, which is enough until a real big-endian model ships.

**Deliverable:** `anamnesis` v0.4.0 — GGUF parsing + dequantization works end-to-end with bit-exact `llama.cpp`-validated output. anamnesis becomes the only Rust crate that can parse *both* safetensors-based (GPTQ/AWQ/BnB/FP8) and GGUF-based quantized models and dequantize everything to BF16. — **PUSH + tag `v0.4.0`**

**New dependencies:** None. The `gguf` feature pulls in `memmap2` (already used by `pth`) and relies on `half` (already mandatory). No third-party GGUF parser in the dependency tree.

### Phase 4 patch: API polish + dogfooding (v0.4.1)

**Goal:** Quick backward-compatible release bundling two GGUF polish items pulled forward from Phase 4.5 and one in-memory safetensors API addition requested by candle-mi dogfooding. Unblocks candle-mi stoicheia module (agnostic `.pth` loading) and gives GGUF users CLI + convenience-method access without waiting for the full `IQ*`/`TQ*`/`MXFP4` kernel work.

**Dogfooding report:** [`docs/dogfooding-feedbacks/in-memory-safetensors-for-candle-mi.md`](docs/dogfooding-feedbacks/in-memory-safetensors-for-candle-mi.md) — candle-mi's stoicheia module needs an in-memory path from `.pth` bytes to safetensors bytes (`VarBuilder::from_buffered_safetensors`). The existing `pth_to_safetensors` writes to disk; the new `pth_to_safetensors_bytes` returns `Vec<u8>`.

- [x] **`pth_to_safetensors_bytes` + `ParsedPth::to_safetensors_bytes`** — in-memory `.pth` → safetensors conversion returning `Vec<u8>` instead of writing to disk. Enables downstream crates (candle-mi) to load `.pth` files without a temp file round-trip. Re-export from `lib.rs`. **5 new tests** (2 unit + 3 integration roundtrip-via-bytes). ~50 lines — **commit**
- [x] **`ParsedGguf::dequantize_tensor` convenience method** — thin wrapper that infers `n_elements` from `shape.iter().try_fold(checked_mul)` and slices the mmap at `data_offset..data_offset + byte_len` before delegating to `dequantize_gguf_to_bf16`. ~40 lines — **commit**
- [x] **GGUF CLI subcommands** — extend `src/bin/main.rs` format detection to recognise the `"GGUF"` magic and add `amn parse model.gguf`, `amn inspect model.gguf`, and `amn remember model.gguf --to bf16 -o out.safetensors`. Dequantizes quantized tensors to `BF16`, passes through non-quantized tensors with their original dtype, reverses GGUF MSB-first shape to safetensors row-major. ~130 lines — **commit** — **PUSH + tag `v0.4.1`**

**Deliverable:** `anamnesis` v0.4.1 — in-memory safetensors API for `.pth`, GGUF CLI subcommands, and `dequantize_tensor` convenience method. ~100 lines total, fully backward compatible. — **PUSH + tag `v0.4.1`**

### Phase 4.5: GGUF Completeness

**Goal:** Close the last remaining coverage gap in GGUF dequantisation so that anamnesis can handle every GGUF block type shipping on HuggingFace today. `v0.4.0` covers the 12 block types that back most mainstream models (`Q4_0`–`Q8_1`, `Q2_K`–`Q8_K`), but a growing fraction of 2025–2026 GGUF uploads use the newer `IQ*` family — `IQ4_XS` in particular has become a common "small but accurate" quant for many Llama 3, Qwen 2.5, and Mistral Nemo variants — plus a handful of models ship `TQ*` or `MXFP4`. Without those kernels, anamnesis rejects real files with `AnamnesisError::Unsupported`.

**Approach:** Port the scalar `dequantize_row_*` reference functions for the remaining block types from `ggml-quants.c` using the same loop-fission template as Phase 4. Most of the work is verbatim translation of the reference kernels plus verbatim copying of the lattice / codebook constant tables that the `IQ*` family uses (`iq2xxs_grid: [u64; 256]`, `iq3s_grid`, `iq1s_grid`, and siblings). Each variant has a distinct block struct layout documented in `ggml-common.h`; the test playbook is unchanged from Phase 4 — hand-constructed single-block fixtures per type, plus bit-exact cross-validation against `llama.cpp` reference output.

- [x] **`IQ4_NL` and `IQ4_XS` dequant kernels** (`src/remember/gguf.rs`, commits `486fa85` + `89abaa6`) — non-linear 4-bit quants (`IQ4_NL` 32-element at 18 B/block, `IQ4_XS` 256-element super-block at 136 B/block) sharing a single 16-entry `kvalues_iq4nl` codebook constant. Ported verbatim from `ggml-quants.c` using the Phase 4 two-pass loop-fission template. Bit-exact (0 ULP) against the `gguf` Python reference on 65 536-element slices from `bartowski/SmolLM2-135M-Instruct-GGUF` (6 new unit tests + 2 new cross-validation tests). First `IQ*` kernels in the crate; `IQ4_XS` is the most widely used member of the `IQ*` family on HuggingFace.
- [x] **`IQ2_XXS`, `IQ2_XS`, `IQ2_S` dequant kernels** (`src/remember/gguf.rs`, commits `cc83de8` + `6385c73`) — three 2-bit super-quants (66 / 74 / 82 B/block) sharing a lattice-codebook design: `IQ2XXS_GRID: [u64; 256]`, `IQ2XS_GRID: [u64; 512]`, `IQ2S_GRID: [u64; 1024]`, plus the `ksigns_iq2xs` / `kmask_iq2xs` sign tables (~14 KB total), all ported verbatim from `ggml-common.h` into a private `iq_grids` submodule at the bottom of the file. Bit-exact (0 ULP) against the `gguf` Python reference on 65 536-element slices from `bartowski/Mistral-7B-Instruct-v0.3-GGUF` (`IQ2_XXS` + `IQ2_XS`) and `bartowski/Qwen2.5-0.5B-Instruct-GGUF` IQ2_M mix (`IQ2_S`). Discovered along the way: `Mistral-7B-v0.3-IQ2_S.gguf` is misleadingly named — it ships `IQ2_XS` + `IQ3_S`, no `IQ2_S`. 7 new unit tests + 3 new cross-validation tests.
- [x] **`IQ3_XXS` and `IQ3_S` dequant kernels** (`src/remember/gguf.rs`, commits `acdcfcd` + `62b4aa3`) — two 3-bit super-quants (98 / 110 B/block) sharing two new codebook grids: `IQ3XXS_GRID: [u32; 256]` (1 KB) and `IQ3S_GRID: [u32; 512]` (2 KB). Both reuse the Phase 4.5 step 2 `write_signed_grid` helper — the combined-grid/signs packing format is shared across every sign-masked `IQ*` kernel. `IQ3_S` introduces the unusual odd-integer scale formula `d × (1 + 2·nibble)` and pairs low/high nibbles of `scales[outer]` across two consecutive sub-blocks. Bit-exact (0 ULP) against the `gguf` Python reference on 65 536-element slices from `bartowski/Mistral-7B-Instruct-v0.3-GGUF` — a single 2.64 GB download (`Mistral-7B-Instruct-v0.3-IQ3_XXS.gguf`) covers both fixtures because that file ships 96 `IQ3_XXS` tensors + 33 `IQ3_S` tensors. 6 new unit tests + 2 new cross-validation tests (3.32× / 4.37× faster than Python reference, release-mode best-of-5).
- [x] **`IQ1_S` and `IQ1_M` dequant kernels** (`src/remember/gguf.rs`, commits `16bd880` + `ecaca11` + `891e4b6`) — two 1-bit super-quants, smallest footprint in the IQ family. `IQ1_S` (50 B/block, top-level `d: f16`, 11-bit grid index, ±`IQ1S_DELTA = 0.125` additive bias); `IQ1_M` (56 B/block, **no top-level `d`** — super-block scale reconstructed from a scattered 16-bit pattern across `scales[8]` reinterpreted as `f16`). Both share the 2048-entry `IQ1S_GRID: [u64; 2048]` codebook of signed `i8` 8-element vectors (~16 KB, the largest single grid in the crate). Inner-loop math is `dl × (grid[j] + delta)` — additive bias instead of IQ2/IQ3's multiplicative ±1 sign — needing the new `write_delta_grid` helper. Bit-exact (0 ULP) against the `gguf` Python reference on 65 536-element slices from `bartowski/Mistral-7B-Instruct-v0.3-GGUF` (separate `IQ1_S.gguf` and `IQ1_M.gguf` files; no shared-file trick available like step 3). 6 new unit tests + 2 new cross-validation tests (13.77× / 7.88× faster than Python reference, release-mode best-of-5 — fastest IQ kernels in the crate).
- [x] **`TQ1_0` and `TQ2_0` dequant kernels** (`src/remember/gguf.rs`, commits `72f8e3c` + `111de18` + `53dd3ab`) — two ternary super-quants (54 / 66 B/block) decoding to `{-d, 0, +d}`. `TQ1_0` uses base-3 packing (5 ternaries per `qs` byte, 4 per `qh` byte) decoded via the `pow3 = [1, 3, 9, 27, 81, 243]` multiplication trick. `TQ2_0` is plain 2-bit packing. New `decode_pow3_ternary` helper alongside the existing `write_signed_grid` / `write_delta_grid` family. **First step using synthetic fixtures** — only ~15 BitNet-derivative GGUFs ship TQ types on HuggingFace, but Python `gguf.quants.quantize()` implements both, so a deterministic synthetic random tensor (seed=42, scale=0.1) is the practical fixture source. Bit-exact (0 ULP) against the `gguf` Python reference. 5 new unit tests + 2 new cross-validation tests (35.59× / 26.31× faster than Python reference, release-mode best-of-5 — fastest GGUF kernels in the crate, beating Q5_0's 31.3× record).
- [x] **`MXFP4` dequant kernel** (`src/remember/gguf.rs`, commits `b87d877` + `acfe8a6` + `89e596d`) — 32-element microscaling FP4 (OCP MX standard, added to `ggml` in 2024), 17 B/block: `e: u8` (E8M0 byte exponent) + `qs[16]` (4-bit packed). Distinct from the `IQ*` / `TQ*` family because it uses a standardised IEEE-like sub-block exponent (`E8M0`) rather than a learned codebook, decoded by a new `e8m0_to_fp32_half` helper that matches `llama.cpp`'s `ggml_e8m0_to_fp32_half` (no NaN guard for `e == 0xFF`, deviating from raw OCP MX spec). 16-entry signed `i8` codebook (`K_VALUES_MXFP4`, storing 2× the OCP E2M1 magnitudes) with the doubling cancelled by the half-scale exponent. Same low/high split-nibble layout as `Q4_0` / `IQ4_NL` via the existing `run_legacy_kernel` runner. Bit-exact (0 ULP) against the `gguf` Python reference on a deterministic synthetic fixture (Python `gguf.quants.quantize()` supports MXFP4 too — same synthetic-fixture path as TQ1_0/TQ2_0 from step 5). 4 new unit tests + 1 new cross-validation test (30.14× faster than Python reference, release-mode best-of-5).
- [x] **Cross-validation extension** — landed incrementally with steps 1–6: every `IQ*` / `TQ*` / `MXFP4` kernel ships its own cross-validation test against the `gguf` Python reference (mirrors `ggml-quants.c`). Bit-exactness contract holds at 0 ULP for all 22 of 22 production kernels — **PUSH + tag `v0.4.2`** (deferred — local CI dry-run completed before push, awaiting user review).

**Deliverable:** `anamnesis` v0.4.2 — anamnesis now dequantises every GGUF block type shipping on HuggingFace. `IQ*`/`TQ*`/`MXFP4` are the last meaningful coverage gap; after this release anamnesis is a drop-in replacement for `llama.cpp`'s CPU dequant path. — **PUSH + tag `v0.4.2`**

**New dependencies:** None. Reuses the `gguf` feature gate, `memmap2`, and `half` from Phase 4.

**Explicitly out of scope:**

- **Big-endian GGUF v3 support** — the parser detects byte-swapped magic and returns `Unsupported`. A reader rewrite can reuse `parse::utils::byteswap_inplace` if demand ever materialises, but no mainstream model is distributed big-endian today. No committed target.
- **CPU SIMD optimisation of pass 2** — cross-cutting optimisation that would benefit FP8, GPTQ, AWQ, and BnB equally, not just GGUF. See [Phase 9: CPU SIMD Pass](#phase-9-cpu-simd-pass).

### Phase 5: Quantization (Lethe)

**Goal:** The opposite direction — take full-precision weights and quantize them. Built on parse (read the source) + lethe (compress). With Phase 4 complete, this enables **cross-format conversion** via the dequantize-then-requantize path (e.g., GPTQ safetensors → BF16 → GGUF Q4_K_M). No Rust tool can do this today.

- [ ] FP8 quantization (`src/lethe/fp8.rs`) — BF16/F32 → FP8 E4M3 with fine-grained block scale factors — **commit**
- [ ] INT8/INT4 quantization — per-channel and group-wise schemes — **commit**
- [ ] `ParsedModel::forget()` public API + `forget` CLI subcommand (alias `quantize`) — **commit**
- [ ] Round-trip validation — quantize then dequantize, measure lethe distance — **commit** — **PUSH**

**Deliverable:** `anamnesis` v0.5.0 — full quantize + dequantize cycle. — **PUSH + tag `v0.5.0`**

### Phase 6: Format Conversion Matrix

**Goal:** Wire the full pipeline — any supported input format to any supported output format in a single command. With Phases 1–5 complete, the core is `parse → remember → forget → write`. Phase 6 adds the CLI for end-to-end conversion, plus output writers for each target format.

**Key conversions unlocked:**

| From | To | Use case |
|------|----|----------|
| GPTQ safetensors | GGUF Q4_K_M | Deploy HuggingFace models in llama.cpp/Ollama |
| AWQ safetensors | safetensors BF16 | Remove quantization for fine-tuning |
| GGUF | safetensors BF16 | Load llama.cpp models in candle/burn |
| safetensors BF16 | GGUF Q4_K_M | Quantize for local inference |
| NPZ | safetensors | Migrate JAX weights to HuggingFace ecosystem |
| PyTorch .pth | safetensors | Migrate legacy PyTorch weights (already supported since v0.3.1) |

- [ ] GGUF output writer — write GGUF files with K-quant types from BF16 input — **commit**
- [ ] `amn convert` CLI subcommand — `amn convert model.safetensors --to gguf-q4km -o model.gguf` — **commit**
- [ ] Cross-format round-trip validation — convert, then convert back, measure distortion — **commit** — **PUSH**

**Deliverable:** `anamnesis` v0.6.0 — any-to-any format conversion. No Rust or Python tool does this today. — **PUSH + tag `v0.6.0`**

### Phase 6.5: Benchmarking & Performance Validation

**Goal:** Establish reproducible performance baselines before the Python bindings expose anamnesis to a much larger audience. Benchmark every dequantization kernel (throughput) and validate memory efficiency claims (peak heap). All benchmarking infrastructure is dev-only — zero impact on the published crate.

**Approach:** Use [criterion](https://github.com/bheisler/criterion.rs) for runtime throughput benchmarks (statistical rigor, regression detection, baselines). Use [dhat-rs](https://github.com/nnethercote/dhat-rs) for peak-heap assertions in integration tests. Both are `[dev-dependencies]` only — bench code in `benches/`, memory tests in `tests/`.

**Runtime benchmarks (`benches/`):**

- [ ] Dequantization throughput — one benchmark per kernel family (FP8, GPTQ 4-bit, AWQ 4-bit, BnB NF4, BnB INT8, GGUF Q4_K), reporting elements/sec on synthetic tensors sized like real model layers (e.g., 4096x11008) — **commit**
- [ ] Parsing throughput — safetensors, NPZ, PTH, GGUF header parsing, reporting MB/s against raw `fs::read` baseline — **commit**

**Peak-memory validation (`tests/`):**

- [ ] Peak-heap assertions for GPTQ/AWQ dequantization — verify lazy precomputation keeps peak heap within `output_size + O(out_features)`, not `output_size + O(num_groups × out_features)` — **commit**
- [ ] Peak-heap assertions for BnB double-quant — verify no intermediate byte-serialization allocation — **commit** — **PUSH**

**New dev-dependencies:** `criterion` (runtime), `dhat` (peak heap). Not compiled into the published crate.

**Deliverable:** Baselines recorded, CI can detect regressions, performance claims in README are backed by reproducible numbers. No version bump — this is infrastructure, not a user-facing release.

### Phase 7: Python Bindings (PyO3)

**Goal:** Expose anamnesis to the Python ecosystem via `pip install anamnesis`. Phases 1–6 give anamnesis the fastest dequantization (2.7–54× vs PyTorch), the fastest NPZ parser (17.7× vs npyz), PyTorch `.pth` parsing, GGUF support, quantization, and any-to-any format conversion — all in pure Rust. Python bindings multiply the audience by ~100×, replacing ad-hoc dequantization scripts across the ML community. By shipping after the format conversion matrix, `pip install anamnesis` includes `convert()` from day one.

**Approach:** Use [PyO3](https://pyo3.rs/) + [maturin](https://github.com/PyO3/maturin) to build a native Python extension. The Python API should mirror the Rust library API closely: `parse()`, `inspect()`, `remember()`, `forget()`, `convert()`, `parse_npz()`. Returns NumPy arrays (via `numpy` interop) or raw bytes. Ships as a wheel on PyPI.

- [ ] PyO3 module scaffold (`src/python.rs`) — feature-gated behind `python` — **commit**
- [ ] `parse_npz()` binding — returns `dict[str, NpzTensor]` with NumPy-compatible arrays — **commit**
- [ ] Safetensors `parse()` + `inspect()` bindings — **commit**
- [ ] `remember()` / `forget()` / `convert()` bindings — dequantize, quantize, and convert from Python — **commit**
- [ ] `maturin` build config, PyPI packaging, CI workflow — **commit**
- [ ] Python test suite — validate against PyTorch reference on the same fixtures — **commit** — **PUSH**

**Deliverable:** `anamnesis` v0.7.0 — `pip install anamnesis` works, with the full conversion matrix exposed. — **PUSH + tag `v0.7.0`**

**New dependencies:** `pyo3`, `numpy` (PyO3 interop). Feature-gated behind `python`.

### Phase 8: Emerging Quantization Formats

**Goal:** Extend coverage to newer quantization formats that currently have zero Rust implementations. Prioritized by ecosystem adoption (HuggingFace model count as of March 2026).

**Landscape (March 2026):**

| Format | HF Models | Ecosystem | Rust Status |
|--------|-----------|-----------|-------------|
| EXL2 | ~4,775 | ExLlamaV2-only (GPU, hobbyist/local) | None |
| HQQ | ~190 | Transformers, vLLM (calibration-free) | mistral.rs only (inference-coupled) |
| AQLM | ~120 | Transformers, vLLM (academic, sub-2-bit) | None |

**Note:** EXL2's successor (EXL3) is emerging. HQQ and AQLM have strong technical merits but very low adoption. These formats are monitored; implementation is deferred until adoption grows or a concrete downstream need arises.

- [ ] EXL2 dequantization — mixed-bitrate (2–8 bpw), ExLlamaV2 binary format — **commit**
- [ ] HQQ dequantization — half-quadratic quantization, 1–8 bit, calibration-free — **commit**
- [ ] AQLM dequantization — additive quantization, sub-2-bit with vector codebooks — **commit**
- [ ] Cross-validation for each format — **commit** — **PUSH**

**Deliverable:** `anamnesis` v0.8.0 — emerging format coverage. — **PUSH + tag `v0.8.0`**

### Phase 9: CPU SIMD Pass

**Goal:** Replace the scalar pass-2 `f32 → BF16` loop in every dequantiser with an explicit AVX2 / NEON vectorised version, giving a 4–8× speedup on the hot path without touching the correctness-verified scalar kernels. This is the first time anamnesis commits to explicit SIMD intrinsics — up to v0.8.0 the project has relied entirely on auto-vectorisation through the loop-fission + `chunks_exact_mut(2)` pattern.

**Why this is a single phase and not per-format work:** Phases 1–4.5 all share the same `f32_bits_to_bf16_bits` pattern (defined once at [src/remember/fp8.rs:101](src/remember/fp8.rs#L101) and reused verbatim by `gptq.rs`, `awq.rs`, `bnb.rs`, and `gguf.rs`). A single `f32x8_to_bf16x8` SIMD helper retrofits every existing dequantiser at once, with one `unsafe` module, one `// SAFETY:` contract, one benchmark harness, and one cross-format round of cross-validation. Splitting the work per format would duplicate the intrinsic surface area across five modules and violate `CONVENTIONS.md`'s "unsafe lives in a single, dedicated module" rule.

**Why now (not earlier):** Auto-vectorisation handles the pass-2 loop reasonably well already — the branch-free `chunks_exact_mut(2)` pattern was chosen specifically to give LLVM a clean target. Phase 9 is triggered when benchmark pressure from real users (enabled by the Phase 6 format conversion CLI and the Phase 7 Python bindings) surfaces kernels where the scalar fallback is a bottleneck. A `TODO(phase4-followup)` comment already sits on [`write_scratch_to_bf16`](src/remember/gguf.rs) flagging the intent.

**Approach:** Add a `simd` feature flag. Under that flag, introduce `src/remember/simd.rs` — a single dedicated module containing `#[target_feature(enable = "avx2")] unsafe fn f32x8_to_bf16x8(...)` and its NEON equivalent. The `write_scratch_to_bf16` helper (currently in `src/remember/gguf.rs` but logically shared across all dequantisers) moves to this module and becomes a runtime dispatcher: `is_x86_feature_detected!("avx2")` → AVX2 path, `std::arch::is_aarch64_feature_detected!("neon")` → NEON path, else fall through to the existing scalar loop. The scalar path stays the canonical correctness reference; the SIMD paths are tested bit-exactly against it via golden-vector cross-checks.

Per [`CONVENTIONS.md`](CONVENTIONS.md)'s accepted-unsafe table, all of the following must hold:

1. `unsafe` lives in a **single, dedicated module** (`src/remember/simd.rs`) — never scattered.
2. Every `unsafe` block carries a `// SAFETY:` comment documenting the invariants (lane count, alignment, target-feature preconditions).
3. The module is gated behind `#[cfg(feature = "simd")]` — users who don't enable the feature get `#![forbid(unsafe_code)]` unchanged.
4. The safe scalar fallback exists and is tested identically to the SIMD path.

- [ ] **`simd` feature flag + `remember::simd` module scaffold** — new `Cargo.toml` entry, `#[cfg(feature = "simd")] pub mod simd;` in `remember/mod.rs`, `cfg_attr(feature = "simd", allow(unsafe_code))` in `lib.rs`, and a new row in the `CONVENTIONS.md` accepted-unsafe table — **commit**
- [ ] **`f32x8_to_bf16x8` AVX2 intrinsic** — 8-wide round-to-nearest-even `f32 → BF16` lane-for-lane equivalent to the scalar `f32_bits_to_bf16_bits`. Bit-exact against the scalar reference (golden-vector test) — **commit**
- [ ] **`f32x4_to_bf16x4` NEON intrinsic** — ARM64 equivalent for Apple Silicon and AWS Graviton. Same bit-exactness requirement as the AVX2 path — **commit**
- [ ] **Runtime dispatch in `write_scratch_to_bf16`** — move the shared helper out of `src/remember/gguf.rs` into `src/remember/simd.rs` (or a new `remember::bf16_writer` shim). Retrofit FP8, GPTQ, AWQ, BnB, and GGUF pass-2 loops to call the shared dispatcher. Delete the per-kernel scalar copies — single source of truth — **commit**
- [ ] **Criterion bench harness** — benchmarks comparing scalar vs SIMD vs PyTorch CPU on the existing FP8/GPTQ/AWQ/BnB/GGUF cross-validation fixtures. Target: ≥ 4× speedup on `BF16`-writing pass 2 for tensors ≥ 1 MB. Publish the numbers in the crate README — **commit**
- [ ] **`copy_to_contiguous` flat-index pass-2 conversion** — replace the cross-iteration `coords[]` carry chain in [`src/parse/pth.rs::copy_to_contiguous`](src/parse/pth.rs) with the stateless `c_i = (flat_idx / stride_i) % shape_i` formula plus specialised fast paths for `ndim ∈ {1, 2, 3}`. Verify with `cargo-show-asm` that the loop emits SIMD strided gathers on AVX2/NEON. Path is rare (<0.1% of `state_dict` files) so this is a polish item, not a hot-path priority. Bit-exact unit tests already cover the function — **commit**
- [ ] **AWQ/GPTQ pass-2 zip-chain audit** — verify whether the four-way `chunks_exact_mut(2).zip(unpacked).zip(zeros).zip(scales)` pattern in [`src/remember/awq.rs`](src/remember/awq.rs) and [`src/remember/gptq.rs`](src/remember/gptq.rs) actually pessimises auto-vectorisation under stable LLVM. If `cargo-show-asm` shows scalar fallback, refactor to an indexed `for i in 0..len { … }` loop over pre-validated equal-length slices (per [`CONVENTIONS.md`](CONVENTIONS.md) "Reconciling bounds checking with vectorization"). Bit-exactness against PyTorch must hold at 0 ULP after refactor — **commit**
- [ ] **Cross-format correctness sweep** — re-run every Phase 1–4.5 cross-validation test with `--features simd` and assert bit-identical output against the scalar path. No format may regress — **commit** — **PUSH + tag `v0.9.0`**

**Deliverable:** `anamnesis` v0.9.0 — `BF16`-writing pass 2 runs 4–8× faster on AVX2 and NEON CPUs, scalar fallback preserved for `forbid(unsafe_code)` users, with cross-kernel consistency (FP8 / GPTQ / AWQ / BnB / GGUF all benefit from the same single-source-of-truth SIMD helper). — **PUSH + tag `v0.9.0`**

**New dependencies:** None at runtime — uses `std::arch` intrinsics directly. Optional dev-dependency on `criterion` for the bench harness.

### Phase 10: Streaming Output

**Goal:** Drop peak heap from `O(total_dequantised_size)` to `O(largest_tensor_BF16)` for whole-model dequantisation paths (`ParsedModel::remember` and the `amn remember model.gguf -o out.safetensors` CLI). Unblocks 70 B+ model conversion on commodity hardware (≤ 32 GB) where the current eager-buffering approach OOMs.

**Background:** The dequantisation kernels already provide streaming entry points — [`dequantize_gguf_blocks_to_bf16`](src/remember/gguf.rs) is `O(one block)` per call, and the `FP8`/`GPTQ`/`AWQ`/`BnB` kernels emit one tensor at a time. The orchestrators do not stream: [`ParsedModel::remember_bf16_inner`](src/model.rs) accumulates every dequantised tensor's `Vec<u8>` into a `dequantized_data: Vec<(name, bytes, shape)>` and then hands all `TensorView`s to `safetensors::serialize_to_file` simultaneously. The same pattern lives in [`src/bin/main.rs::run_remember_gguf`](src/bin/main.rs). The `safetensors` 0.4 crate's writer **already streams tensor bodies** (one `BufWriter::write_all` per tensor inside `serialize_to_file`) — the eager buffering is ours, not the crate's.

**Approach:** Implement a custom `View` impl whose `data()` lazily dequantises a single tensor on demand, paired with an iterator that yields `(name, LazyView)` derived from the parsed model. The crate's `prepare()` step only needs `data_len()` / `dtype()` / `shape()` for the header — all deterministic and cheap. Hand the iterator to `safetensors::serialize_to_file`; the writer pulls bytes per tensor and drops them after each `write_all`. Peak heap drops to `O(largest_tensor_BF16)` — for a 70 B model's biggest layer (~500 MB BF16) this fits on commodity hardware. True per-block O(1) streaming is **out of scope** for Phase 10 — it would require a custom safetensors writer (the `View` trait returns a single `Cow<[u8]>`, not a stream) and the per-tensor variant is sufficient for every model size in current circulation.

- [ ] **`LazyDequantView<'a>` type** in `src/remember/streaming.rs` — implements `safetensors::tensor::View`, captures a borrow into the parsed model + the per-tensor metadata needed to dispatch into the right kernel, and returns `Cow::Owned(dequantised_bytes)` from `data()` on demand — **commit**
- [ ] **`ParsedModel::remember_streaming`** — yields lazy views to `safetensors::serialize_to_file`. Rewrite `remember()` to delegate to `remember_streaming()` so existing call sites pick up the win automatically — **commit**
- [ ] **GGUF CLI streaming path** — refactor `run_remember_gguf` in `src/bin/main.rs` to drop the `tensor_data: Vec<(...)>` accumulator in favour of a lazy iterator. Same `LazyDequantView` infrastructure — **commit**
- [ ] **Peak-heap regression tests via `dhat-rs`** — assert peak heap ≤ `2 × largest_tensor_BF16 + small_constant` on a representative 7 B fixture (FP8 + GGUF). Reuses Phase 6.5 dhat-rs infrastructure — **commit**
- [ ] **Cross-format correctness sweep** — re-run every Phase 1–4.5 cross-validation test against the streaming output path; output bytes must be byte-identical to the eager path on the same input — **commit** — **PUSH + tag `v0.10.0`**

**Deliverable:** `anamnesis` v0.10.0 — peak heap drops from `O(model_size)` to `O(largest_tensor)` for whole-model dequantisation. 70 B+ models become convertible on commodity hardware. All existing public APIs preserved (`remember()` becomes a thin wrapper). — **PUSH + tag `v0.10.0`**

**New dependencies:** None at runtime. Reuses the Phase 6.5 `dhat-rs` dev-dependency for peak-heap assertions.

**Out of scope:**

- **True per-block O(1) streaming** — requires a custom safetensors writer (the `View::data()` method returns `Cow<[u8]>`, not a stream). Per-tensor lazy dequant covers every plausible model size; per-block remains a Future Directions item if real-world peak-heap measurements after Phase 10 ever show insufficient headroom.
- **`WASM32` linear-memory adaptation** — WASM32's 4 GB memory cap is independent of `usize` typing; supporting it requires WASM64 or a fundamentally different I/O model. Listed under Future Directions.

### Future Directions

The following are potential extensions beyond Phase 10, listed for context. They will be promoted to full phases when a concrete need arises.

- **Model surgery** — extract specific layers, merge LoRA adapters with base weights at the file level, split/shard for distributed loading
- **Quantization quality analysis** — per-layer distortion reports, optimal mixed-precision selection, diff two quantized versions of the same model
- **WASM target** — compile anamnesis to WASM for browser-based model inspection (drop a safetensors file, see tensor layout and quantization scheme instantly)
- **GPU-accelerated dequantization** — compute shader (wgpu/Vulkan) kernels for dequantization hot loops, for cases where the Phase 9 CPU SIMD pass is still insufficient (e.g., bulk 70 B-parameter conversions on a workstation with an idle GPU)

---

## 4. Key Design Decisions

### Framework-agnostic output

anamnesis outputs standard safetensors files (for dequantization) and raw byte arrays with shape/dtype metadata (for NPZ). It never depends on candle, burn, or tch. Any framework can consume its output.

### Parse-first architecture

Every code path begins with parsing. The `parse/` module is the foundation; `remember/`, `lethe/`, and `inspect` are built on top. This is not incidental — it reflects the design principle that you cannot remember what you have not first parsed.

### Feature gates per format

Each quantization scheme and container format is behind its own feature flag. Users pay only for what they need. Default: `fp8` only.

### No GPU dependency

All operations are pure CPU. FP8 dequantization is bit manipulation, not matrix multiplication. This keeps the crate lightweight and universally deployable.

### SIMD strategy

Conversion loops process billions of elements and are embarrassingly parallel
(each element is independent). The strategy is layered:

1. **Phase 1: SIMD-friendly scalar code.** Write loops that follow the
   CONVENTIONS.md SIMD-friendly rules (contiguous slices, no branches,
   `chunks_exact`, hoisted invariants, separate input/output buffers).
   Verify auto-vectorization with `cargo-show-asm`. This requires **no
   `unsafe`**, no nightly, no extra dependencies, and builds on stable Rust.
   The compiler typically emits AVX2 on x86-64 (Intel + AMD) and NEON on ARM.

2. **If benchmarks demand more: hand-written intrinsics** behind a `simd`
   feature gate. `#[target_feature(enable = "avx2")]` + runtime
   `is_x86_feature_detected!` dispatch, with the scalar path as fallback.
   One dedicated module (`src/remember/simd.rs`), tested against the scalar
   reference. Requires `unsafe` (see CONVENTIONS.md `// SAFETY:` rules).

3. **Not pursued: `std::simd` (portable SIMD).** Requires nightly Rust, which
   would virally infect downstream crates (candle-mi, hf-fetch-model). If
   `std::simd` stabilizes, it becomes the preferred path and replaces both
   the scalar and intrinsic implementations.

Coverage: AVX2 covers all Intel + AMD CPUs since ~2015. NEON covers Apple
Silicon and ARM servers. The scalar fallback covers everything else.

### Error type

A single `AnamnesisError` enum with variants per failure category:
- `Parse { reason }` — format decoding failures (including `NPZ`/`NPY` and `ZIP` errors)
- `Unsupported { format, detail }` — recognized but unimplemented format
- `Io(std::io::Error)` — file system errors

---

## 5. Relationship to Other Projects

### hf-fetch-model

Download crate. Depends on anamnesis for `--dequantize` and `download_and_parse_npz()`. anamnesis provides the format intelligence; hf-fetch-model provides the network I/O. See hf-fetch-model v0.8.0 roadmap.

### candle-mi

MI framework. anamnesis intersects with candle-mi in two ways:

1. **NPZ migration.** candle-mi contained a tightly-coupled NPZ parser for Gemma Scope SAE weights. Phase 3 of anamnesis replaces it. Migration design: `candle-mi/design/migrate-npz-to-anamnesis.md`. `anamnesis` v0.3.0+ provides `parse_npz()` with a `From<AnamnesisError>` bridge to `MIError`. Target: candle-mi v0.1.5.

2. **Auto-config extensions.** The FP8 test models chosen for anamnesis Phase 1 validation introduce two new architectures that candle-mi should support:
   - **`exaone4`** (`LGAI-EXAONE/EXAONE-4.0-1.2B-FP8`) — LLaMA-like with alternating sliding window / full attention ("LLLG" pattern). Close to Gemma 2's alternating scheme. Requires a new `parse_exaone4()` in `config.rs`.
   - **`qwen3`** (`Qwen/Qwen3-1.7B-FP8`) — extends `qwen2` with QK LayerNorm (`q_norm` / `k_norm` per layer). Requires a new `parse_qwen3()` in `config.rs` and handling the extra norm tensors in the forward pass.

   These extensions benefit candle-mi independently of anamnesis — both model families are mainstream and worth supporting regardless.

### candle

HuggingFace's Rust ML framework. anamnesis outputs standard safetensors that candle can load directly. No dependency in either direction — anamnesis reads the safetensors format, not candle's tensor types.
