# ML-KEM Linux Keygen Loss Closure Plan

Created: 2026-06-30

Goal: close the current Linux `mlkem768/keygen` and `mlkem1024/keygen` fastest-external
losses without vendoring more assembly. Owned Rust intrinsics, owned assembly,
platform dispatch, and large implementation effort are all allowed when the data
supports them.

The target rows are from Linux CI run
[#28461694270](https://github.com/loadingalias/rscrypto/actions/runs/28461694270),
commit `1bcc2d19cf7b2df93ad3e814a1763ab9ce7e84b4`.

| Primitive/op | Rows | W/T/L | Geomean | Median | Main pressure |
| --- | ---: | ---: | ---: | ---: | --- |
| `mlkem1024/keygen` | 9 | 1/0/8 | 0.82x | 0.78x | `libcrux` 4, `aws-lc-rs` 4 |
| `mlkem768/keygen` | 9 | 1/1/7 | 0.92x | 0.89x | `aws-lc-rs` 4, `libcrux` 3 |

## Baseline Loss Map

Ratios are `fastest_external_time / rscrypto_time`; higher is better.

| Platform | `mlkem768/keygen` | Fastest external | `mlkem1024/keygen` | Fastest external |
| --- | ---: | --- | ---: | --- |
| AMD Zen4 | 0.894x | `libcrux` | 0.763x | `libcrux` |
| AMD Zen5 | 0.902x | `libcrux` | 0.779x | `libcrux` |
| AWS Graviton3 | 0.817x | `aws-lc-rs` | 0.733x | `aws-lc-rs` |
| AWS Graviton4 | 0.771x | `aws-lc-rs` | 0.695x | `aws-lc-rs` |
| IBM Power10 | 0.997x | `libcrux` | 0.940x | `libcrux` |
| IBM z16/s390x | 1.360x | `libcrux` | 1.216x | `libcrux` |
| Intel Ice Lake | 0.859x | `libcrux` | 0.747x | `libcrux` |
| Intel Sapphire Rapids | 0.930x | `aws-lc-rs` | 0.811x | `aws-lc-rs` |
| RISE RISC-V | 0.859x | `aws-lc-rs` | 0.812x | `aws-lc-rs` |

The loss classes are:

- x86_64: eight losses across Zen4, Zen5, Ice Lake, and Sapphire Rapids.
- Linux AArch64: four losses across Graviton3 and Graviton4.
- RISC-V: two losses.
- Power10: one near-tie `mlkem1024/keygen` loss.
- s390x: already wins; it is a guardrail, not an optimization target.

## Hard Constraints

- Do not vendor more ML-KEM assembly.
- Do not restore deleted external-derived ML-KEM NTT assembly.
- Do not write end-to-end keygen assembly unless all narrower paths fail with
  measured evidence.
- Do not promote a candidate that improves one platform by creating a fastest-row
  loss on `mlkem512`, encapsulation, decapsulation, or s390x.
- Do not spend paid Graviton/RISC-V/Power CI runs on guesses. A remote run needs
  local diagnostic evidence or an architecture-specific code change that cannot
  be validated locally.
- Public matrix rejection-sampling branches are allowed to depend on public XOF
  bytes. Secret-dependent control flow remains forbidden.

## Current Technical Read

The old AArch64 forward NTT problem is closed. The current losses sit in keygen's
public-matrix generation and accumulation path.

Current keygen shape:

1. `pke_keygen_expanded` samples `s` and `e`.
2. It NTTs both vectors.
3. It runs `keygen_matrix_accumulate`.
4. It converts from product domain.
5. It encodes the public and private key material.

The diagnostic data says matrix generation/accumulation dominates the losing
rows. The strongest structural difference versus AWS-LC and libcrux is the
initial public-matrix sampler:

- AWS-LC/libcrux shape: initialize XOF lanes, squeeze the first three SHAKE128
  rate blocks per polynomial, parse that contiguous buffer as one capped
  rejection pass, then handle rare tail blocks.
- Current rscrypto shape: parse one rate block at a time. AArch64 has owned
  one-block rejection assembly, but bounded tail parsing goes through a
  scratch/copy path. x86_64 matrix sampling still falls back to scalar rejection
  parsing.

This plan attacks that difference directly.

## 2026-07-01 Live Hypothesis

The next attempt is not another end-to-end keygen rewrite. The previous passes
already falsified the broad theories:

- AArch64 forward NTT ownership was fixed, but Graviton keygen losses remained.
- Fused/materialized matrix strategy switching did not close Linux AArch64.
- Cached row multiplication and `polyvec_mulcache` are too small to explain the
  ML-KEM-1024 keygen gap.
- K=4 quad row scheduling wins on Apple Silicon but regresses Graviton, so it is
  not the Linux closure path.
- A scalar bounded-tail parser was slower locally; do not revive it as
  production code.

The live hypothesis is narrower:

1. For AArch64 Linux, the missing primitive is an owned initial-three-block
   `SampleNTT` parser that accepts one contiguous `3 * SHAKE128_RATE_BYTES`
   buffer and writes at most 256 coefficients directly to the destination. This
   removes the current third-block scratch/copy cap in the predictable
   two-full-blocks-plus-bounded-third-block matrix sampler shape.
2. For x86_64 Linux, the missing primitive is an owned SIMD public-matrix
   rejection parser. The existing x86 arithmetic kernels are not enough when
   public matrix sampling is still scalar.
3. If those two parser paths improve diagnostics but do not close public keygen,
   the next target is ML-KEM-private batched SHAKE128/Keccak for matrix
   sampling, not more matrix schedule reshuffling.
4. RISC-V and Power10 remain separate campaigns. The parser work may help, but
   the current RISC-V phase data says NTT and matrix costs are both material, so
   claiming all-target closure from AArch64/x86 parser work would be dishonest.

The first implementation step is diagnostic-only: add a structured
initial-three-block parser boundary and equivalence tests. Production promotion
requires lower-level row-sample movement first; no new assembly is promoted
from end-to-end keygen noise alone.

## 2026-07-01 Initial Implementation Slice

Implemented:

- `diag` parser rows for a structured initial-three-block SampleNTT boundary:
  - `mlkem-sampler-phases/k=4/triple-initial-3blocks-parse`
  - `mlkem-sampler-phases/k=4/triple-initial-3blocks-seeded`
- A safe Rust reference parser for `3 * SHAKE128_RATE_BYTES` capped parsing.
- Tests proving the initial-three-block parser prefix and tail-completed output
  match the existing SampleNTT path for all K=4 matrix coordinates across
  deterministic seeds.
- An owned AArch64 bounded one-block rejection parser ABI for macOS and Linux:
  - `rscrypto_mlkem_rej_uniform_block_bounded_aarch64_apple_darwin`
  - `rscrypto_mlkem_rej_uniform_block_bounded_aarch64_linux`
- Production AArch64 bounded tail parsing now writes directly into the
  destination instead of parsing into a 112-candidate stack scratch and copying
  the accepted prefix.

Local Apple Silicon quick signal:

| Row | Before | After | Read |
| --- | ---: | ---: | --- |
| `k=4/triple-parse-3-blocks` | 2.173 us | 2.116 us | directionally faster; not significant |
| `k=4/triple-initial-3blocks-parse` | 2.518 us | 2.523 us | scalar reference is not production-worthy |
| `k=4/triple-initial-3blocks-seeded` | 2.361 us | 2.307 us | directionally faster; not significant |
| `k=4/triple-third-tail-after-two` | 1.629 us | 1.625 us | flat |
| `k=4/row-sample-counted` | 7.361 us | 7.202 us | about 2.9% faster; p=0.06 |
| `mlkem1024/keygen/rscrypto` | n/a in same saved bench set | 12.6 us | needs Linux confirmation |

Decision: keep the bounded AArch64 ABI for targeted Linux validation. It is
correct by local equivalence tests, compiles for `aarch64-unknown-linux-gnu`,
and moves local row sampling in the right direction without a clear regression.
It is not enough to declare keygen closed. The next paid run should be narrow:
G3/G4 sampler rows plus `mlkem1024/keygen`, with `quick=true`.

## 2026-07-01 Linux Initial-Three-Block Diagnostic Slice

Implemented:

- An owned Linux AArch64 diagnostic parser ABI:
  - `rscrypto_mlkem_rej_uniform_3blocks_aarch64_linux(out, input) -> usize`
  - `input` is exactly three contiguous SHAKE128 rate blocks.
  - `out` has capacity for one full 256-coefficient polynomial.
  - the parser caps writes internally at 256 accepted coefficients.
- Diagnostic Rust wiring for the real row-sampling shape:
  - K=3 `materialized-row-sample-initial-3blocks`
  - K=4 `materialized-row-sample-initial-3blocks`
  - triple-lane seeded initial-three-block sampling now uses a shared triple
    SHAKE schedule instead of three independent single-lane XOFs.
- Forced `windowed-through-matrix-accumulate` rows remain diagnostic only. They
  are not production-selected because local evidence did not justify it.
- The Darwin three-block ABI was intentionally not promoted. macOS keeps the
  existing owned one-block parser; the new three-block symbol is Linux-only
  until the Linux runner proves the hypothesis.

Validation run:

```text
cargo fmt --all -- --check
cargo check --features 'ml-kem'
cargo check --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth
cargo check --target aarch64-unknown-linux-gnu --features 'ml-kem,diag' --lib
cargo test --features 'ml-kem,diag' sample_ntt_ -- --nocapture
cargo test --features 'ml-kem,diag' materialized_k -- --nocapture
cargo test --test mlkem_acvp --features 'ml-kem' -- --nocapture --test-threads=1
cargo test --test mlkem_ops --features 'ml-kem' -- --nocapture --test-threads=1
```

Local Apple Silicon quick signal is wiring-only because macOS does not use the
new Linux three-block parser. The scalar diagnostic row is slower locally:

| Row | Quick result | Read |
| --- | ---: | --- |
| `k=3/materialized-row-sample` | 3.647 us | baseline row wiring works |
| `k=3/materialized-row-sample-initial-3blocks` | 4.774 us | scalar parser is not production-worthy |
| `k=4/materialized-row-sample` | 5.400 us | baseline row wiring works |
| `k=4/materialized-row-sample-initial-3blocks` | 8.854 us | scalar parser is not production-worthy |
| `k=4/triple-initial-3blocks-seeded` | 1.574 us | shared triple schedule is runnable |

Decision rule for the next Linux quick run:

- If Linux AArch64 `materialized-row-sample-initial-3blocks` beats
  `materialized-row-sample` for K=3 and K=4 by at least 3%, production promotion
  is credible.
- If it is flat or slower, stop parser work. The next real target is
  ML-KEM-private batched SHAKE128/Keccak for matrix sampling, or the separate
  x86_64 public-matrix parser campaign.

## 2026-07-02 Graviton Parser Decision

Runs
[`#28563670668`](https://github.com/loadingalias/rscrypto/actions/runs/28563670668)
and
[`#28564250055`](https://github.com/loadingalias/rscrypto/actions/runs/28564250055)
falsified the row-structure candidates:

- `windowed` row sampling is noise-level: K=4 moved about 0.27% on Graviton3
  and 0.09% on Graviton4.
- `fused` K=4 matrix sampling is slower than materialized on both Graviton
  runners.
- `initial-3blocks` is flat or slower. The copy-to-contiguous shape should not
  be promoted.
- `quad` row sampling is slower than triple on Graviton3/4.

The useful signal is lower level. The current production triple parser is not a
true triple parser on Linux: it calls the owned one-block parser three times.
Latest K=4 sampler medians:

| Row | Graviton3 | Graviton4 | Read |
| --- | ---: | ---: | --- |
| `triple-squeeze-3-blocks` | 1.939 us | 1.783 us | Keccak/squeeze is the largest single bucket |
| `triple-parse-3-blocks` | 1.263 us | 1.028 us | parser overhead is large enough to attack |
| `triple-first-two-blocks` | 1.598 us | 1.448 us | first two unbounded blocks matter |
| `triple-third-tail-after-two` | 1.270 us | 1.162 us | bounded tail still matters |
| `materialized-row-sample` K=4 | 13.133 us | 11.578 us | row-level target |

Implementation checkpoint:

- Added owned Linux AArch64 triple-block parser ABIs:
  - `rscrypto_mlkem_rej_uniform_triple_block_aarch64_linux`
  - `rscrypto_mlkem_rej_uniform_triple_block_bounded_aarch64_linux`
- The unbounded ABI parses three independent SHAKE128 rate blocks in one call
  and returns packed per-lane counts.
- The bounded ABI takes a three-element capacity array and handles the third
  block / tail shape without three separate setup paths.
- Production `sample_ntt_triple_block_neon_ptrs` now uses those ABIs on Linux
  instead of calling the one-block parser three times.
- Added AArch64 Linux scalar-reference tests for both packed-count triple
  symbols, including zero-cap bounded lanes.

Expected result: this should reduce `triple-parse-3-blocks`,
`triple-first-two-blocks`, and `triple-third-tail-after-two`. It probably will
not close all Graviton keygen losses by itself; the full K=4 public gap still
points at Keccak/squeeze plus parser work. If the triple parser moves the row
but public keygen remains short, the next implementation is an owned
ML-KEM-private x3 Keccak/sampler schedule, not another row-order experiment.

Required quick validation:

```text
RUNNERS: g3,g4
FILTERS:
  ^mlkem-sampler-phases/k=4/(triple-parse-3-blocks|triple-first-two-blocks|triple-third-tail-after-two|triple-squeeze-3-blocks)
  ^mlkem-keygen-matrix/k=(3|4)/materialized-row-sample
  ^mlkem-matrix-sample/k=(3|4)/triple
  ^(mlkem768|mlkem1024)/keygen/(rscrypto|aws-lc-rs)
QUICK: true
```

## 2026-07-02 AArch64 Fused Chunk-Loop Slice

Run
[`#28612736993`](https://github.com/loadingalias/rscrypto/actions/runs/28612736993)
validated the non-x86 materialized dispatch cleanup at commit
`8a8e5b778ad14a6aed4331ba8d0da6864d19cce9`. Power10 closed and RISC-V moved,
but Graviton did not:

| Platform | K=3 default matrix | K=3 fused matrix | K=4 default matrix | K=4 fused matrix |
| --- | ---: | ---: | ---: | ---: |
| Graviton3 | 12.683 us | 16.976 us | 20.800 us | 22.578 us |
| Graviton4 | 11.033 us | 14.196 us | 17.961 us | 19.264 us |

The public default is correctly materialized, but the fused path remained too
slow to promote. The next implementation attacks a real low-level cost inside
that fused path instead of adding another row schedule:

- Replace the Linux AArch64 fused SampleNTT product path's per-16-coefficient
  Rust NEON chunk multiply with an owned assembly chunk loop:
  `rscrypto_mlkem_basemul_accumulate_chunks_aarch64_linux`.
- The new ABI processes a caller-provided count of contiguous 16-coefficient
  chunks, loading Montgomery constants once per SHAKE block instead of once per
  chunk.
- Remove the stale one-chunk Linux symbol; tests exercise the production
  multi-chunk wrapper with chunk counts up to the fused block maximum.
- Keep production AArch64 keygen dispatch materialized until targeted Graviton
  data proves fused is faster. This change is a promotion candidate, not a
  public dispatch switch.

Local verification for the implementation slice:

```text
cargo check --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --lib
cargo check --target aarch64-unknown-linux-gnu --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --lib
cargo check --target aarch64-unknown-linux-gnu --no-default-features --features 'ml-kem' --lib
cargo check --target aarch64-unknown-linux-gnu --no-default-features --features 'ml-kem,diag' --lib
cargo test --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' auth::mlkem::portable::tests:: --lib
```

The local AArch64 Linux test binary cannot be built on this macOS host because
test/dev dependencies require `aarch64-linux-gnu-gcc` for C code in
`alloca`/`aws-lc-sys`. The library target, including the new ASM ABI, compiles.

Required quick validation:

```text
RUNNERS: g3,g4
PRIMITIVES/SELECTORS: mlkem768,mlkem1024,mlkem-keygen-matrix
FILTERS:
  ^mlkem(768|1024)/keygen/(rscrypto|aws-lc-rs)$
  ^mlkem-keygen-matrix/k=(3|4)/(default-through-matrix-accumulate|fused-through-matrix-accumulate|materialized-through-matrix-accumulate|windowed-through-matrix-accumulate|materialized-row-sample|materialized-row-multiply)$
QUICK: true
```

Promotion rule: switch AArch64 Linux production dispatch to fused only if
`fused-through-matrix-accumulate` beats materialized/default for K=3 and K=4 on
both Graviton runners. If fused improves but remains behind materialized, keep
the owned chunk loop only as groundwork and move to the next lower-level
sampler/Keccak target.

## Success Criteria

Final success:

1. Primary target: the 15 current Linux losses become wins.
2. `mlkem1024/keygen`: W/T/L becomes `9/0/0`, with Linux fastest-external
   geomean >= `1.05x`.
3. `mlkem768/keygen`: W/T/L becomes `9/0/0`, with Linux fastest-external
   geomean >= `1.05x`.
4. Minimum closure checkpoint: both target rows have zero losses. This is not
   the final goal; it only means the loss fire is out and the remaining tie rows
   need a win-focused follow-up.
5. `mlkem512/keygen`, `mlkem512/encapsulate`, `mlkem512/decapsulate`,
   `mlkem768/encapsulate`, `mlkem768/decapsulate`, `mlkem1024/encapsulate`, and
   `mlkem1024/decapsulate` do not lose existing fastest-external wins or ties.
6. s390x remains a win for both target rows.
7. All promoted assembly is rscrypto-owned under `docs/tasks/asm.md`.

Per-candidate promotion threshold:

- Public target rows move by at least 3% on local or directly targeted Linux
  benchmark evidence.
- The candidate explains its win in a lower-level diagnostic row, not just a
  noisy end-to-end keygen row.
- No target platform regresses by more than 1% unless the regression is traded
  for enough wins to improve the final W/T/L target and has a follow-up fix.

## Measurement Setup

Before coding:

1. Freeze the current baseline artifacts.
   Verify: `benchmark_results/OVERVIEW.md` still points at run `#28461694270`
   and commit `1bcc2d19cf7b2df93ad3e814a1763ab9ce7e84b4`.
2. Re-run local correctness once so later failures are attributable.
   Verify: `cargo test --features 'ml-kem,diag' auth::mlkem::portable::tests:: -- --nocapture`.
3. Re-run local targeted benches on the development machine.
   Verify:
   ```text
   cargo bench --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' \
     --bench auth \
     -- '^mlkem-sampler-phases/|^mlkem-keygen-matrix/|^mlkem-matrix-sample/|^(mlkem512|mlkem768|mlkem1024)/keygen/' \
     --quick
   ```
4. Extract the architecture loss table after every remote run.
   Verify: update this file or `benchmark_results/OVERVIEW.md` with the new run
   id, commit, per-platform target rows, and W/T/L delta.

## Step 1: Diagnostic Three-Block SampleNTT Path

Objective: prove or falsify that the initial three-block sampler is the missing
primitive before writing production assembly.

Implementation scope:

1. Add a `diag`-only one-polynomial helper:
   `diag_sample_ntt_initial_3blocks_into(rho, j, i, out)`.
   It must squeeze exactly three SHAKE128 rate blocks for one public matrix
   polynomial, parse the contiguous `3 * SHAKE128_RATE_BYTES` buffer in one
   capped pass, and return:
   - filled coefficient count after the initial pass,
   - number of accepted candidates scanned,
   - whether a tail block was needed,
   - a digest of the output polynomial.
2. Add `diag` row variants for K=3 and K=4 materialized sampling:
   - `mlkem-sampler-phases/k=3/initial-3blocks-parse`
   - `mlkem-sampler-phases/k=3/initial-3blocks-tail`
   - `mlkem-sampler-phases/k=4/initial-3blocks-parse`
   - `mlkem-sampler-phases/k=4/initial-3blocks-tail`
   - `mlkem-keygen-matrix/k=3/materialized-row-sample-3blocks`
   - `mlkem-keygen-matrix/k=4/materialized-row-sample-3blocks`
3. Keep production dispatch unchanged in this step.
4. Use structured parsing, not ad hoc byte slicing scattered through the call
   tree. The parser boundary should be reusable by AArch64 and x86 backends.

Correctness verification:

1. Three-block diagnostic output matches existing `sample_ntt_into` for fixed
   rho/index pairs.
   Verify: unit tests cover all 16 K=4 matrix coordinates and all 9 K=3
   coordinates.
2. Tail behavior matches the existing block-loop parser for seeds that finish in
   three blocks and seeds that need a fourth block.
   Verify: deterministic test vectors include both cases.
3. ACVP/keygen coverage remains unchanged.
   Verify:
   ```text
   cargo test --test mlkem_acvp --features ml-kem -- --nocapture
   cargo test --test mlkem_ops --features ml-kem -- --nocapture
   ```
4. Miri remains on the portable path.
   Verify: `just test-miri` or the existing narrower Miri recipe if the full
   recipe is too expensive.

Performance verification:

1. Local sampler rows show whether contiguous three-block parsing is worth
   pursuing.
   Verify: compare `initial-3blocks-parse`, `initial-3blocks-tail`,
   `triple-parse-3-blocks`, `triple-third-tail-after-two`, and
   `materialized-row-sample`.
2. Promotion gate for Step 2/3:
   - AArch64: the diagnostic must reduce or clearly isolate
     `triple-third-tail-after-two` and row-sample cost.
   - x86_64: the diagnostic must show scalar parsing is a material share of
     row-sample cost.

Rejection criteria:

- If three-block diagnostics do not move or explain row-sample time, stop parser
  work and jump to Step 4 with a written note.

## Step 2: Owned AArch64 Multi-Block Rejection Parser

Objective: close the Graviton3/4 losses by replacing the current one-block
bounded parser shape with an owned capped parser for the first three SHAKE128
blocks.

Target rows:

- Graviton3 `mlkem768/keygen`: 0.817x.
- Graviton3 `mlkem1024/keygen`: 0.733x.
- Graviton4 `mlkem768/keygen`: 0.771x.
- Graviton4 `mlkem1024/keygen`: 0.695x.

Implementation scope:

1. Add an owned AArch64 ABI, preferably:
   ```text
   rscrypto_mlkem_rej_uniform_3blocks_aarch64_linux(out, input) -> usize
   rscrypto_mlkem_rej_uniform_3blocks_aarch64_apple_darwin(out, input) -> usize
   ```
   where `input` points to `3 * SHAKE128_RATE_BYTES` bytes and `out` has room
   for 256 `u16` coefficients.
2. If a generic buffer ABI is simpler and faster, use:
   ```text
   rscrypto_mlkem_rej_uniform_buf_capped_aarch64_*(out, input, buflen) -> usize
   ```
   with `buflen` restricted to multiples of 24 or 48 and initially called only
   with `3 * SHAKE128_RATE_BYTES`.
3. The parser must cap writes at 256 coefficients internally. It must not require
   the Rust wrapper to route the third block through `sample_ntt_block_asm_bounded`.
4. Reuse the existing owned one-block parser only as local design context. Do not
   adapt AWS-LC, mlkem-native, or other external assembly.
5. Keep the ABI public-data only. Branches and write counts depend on public
   matrix XOF bytes.
6. Wire the new parser behind `diag` first:
   - `sample_ntt_initial_3blocks_aarch64_into`
   - `sample_ntt_triple_initial_3blocks_into`
   - `sample_ntt_quad_initial_3blocks_into`
7. Promote production AArch64 only after diagnostic rows win on both local and
   targeted Linux data.

Correctness verification:

1. Assembly wrapper documents the ABI, aliasing, feature gate, and public timing
   boundary.
   Verify: `// SAFETY:` comments state numbered invariants.
2. Unit tests compare new parser output with scalar parser over:
   - all K=3 matrix coordinates,
   - all K=4 matrix coordinates,
   - seeds requiring exactly three blocks,
   - seeds requiring four or more blocks,
   - unaligned input pointers if the ABI permits them.
3. Existing ML-KEM vector tests pass.
   Verify:
   ```text
   cargo test --features 'ml-kem,diag' auth::mlkem::portable::tests:: -- --nocapture
   cargo test --test mlkem_acvp --features ml-kem -- --nocapture
   cargo test --test mlkem_ops --features ml-kem -- --nocapture
   ```
4. Assembly ledger is updated.
   Verify: `docs/tasks/asm.md` lists the new files as rscrypto-owned with
   provenance.

Performance verification:

1. Local AArch64/macOS diagnostic rows move in the expected direction.
   Verify: `materialized-row-sample-3blocks` beats current row sample without
   regressing K=3.
2. Graviton targeted run covers only ML-KEM keygen plus sampler diagnostics.
   Verify:
   ```text
   PRIMITIVES/SELECTORS: mlkem512,mlkem768,mlkem1024
   FILTERS: ^mlkem-sampler-phases/|^mlkem-keygen-matrix/|^(mlkem512|mlkem768|mlkem1024)/keygen/
   ```
3. Promotion gate:
   - Graviton3 and Graviton4 `mlkem768/keygen` become ties or wins.
   - Graviton3 and Graviton4 `mlkem1024/keygen` improve by at least 10%; if not
     enough to close, Step 4 must identify the remaining phase.

Rollback/rejection:

- If the parser improves diagnostics but not public keygen, keep it behind
  `diag` and inspect matrix multiply/from-product-domain interaction.
- If the parser regresses Graviton but wins Apple Silicon, do not promote it to
  Linux. Keep the existing platform split.

2026-07-02 result:

- The buffered initial-three-block parser did not justify promotion. Run
  `28551783178` showed `k=4/materialized-row-sample-initial-3blocks` slightly
  slower than the production row sampler on both Graviton3
  (`13.099us -> 13.229us`) and Graviton4 (`11.546us -> 11.581us`).
- The triple one-block parser was a valid cleanup but not a public win. Run
  `28565902237` kept row sampling essentially flat while only moving the
  parser-only diagnostic by roughly 1-2%.
- The generic owned Keccak x3 assembly was rejected and removed. Run
  `28597139827` regressed the public target rows badly: Graviton3
  `mlkem768/keygen` `17.010us -> 21.207us`, Graviton3
  `mlkem1024/keygen` `26.881us -> 34.032us`, Graviton4
  `mlkem768/keygen` `14.893us -> 17.947us`, and Graviton4
  `mlkem1024/keygen` `23.395us -> 28.553us`.
- The narrow Rust-level ML-KEM-private XOF/sampler boundary was also rejected
  locally before promotion. Against clean `4887c16`, it moved Apple Silicon
  `mlkem1024/keygen` from `16.561us` to `16.771us` and
  `k=4/materialized-row-sample` from `7.499us` to `7.606us`. Delete it rather
  than send another Graviton run.
- The remaining plausible sampler/XOF work must be lower level than reader
  bookkeeping: a fused ML-KEM matrix-sampler kernel that keeps the SHAKE state
  schedule and rejection parser together, or a profile proving a different
  phase dominates. Do not revive the buffered three-block parser, the direct
  Rust rate-block helper, or a generic Keccak permutation rewrite unless new
  data reverses these results.

## Step 3: Owned x86_64 SIMD Matrix Sampler

Objective: close the eight x86_64 target losses with an owned SIMD first-three
block matrix sampler. Start with Rust intrinsics; use assembly only if the
intrinsics version proves the algorithm and leaves obvious codegen on the table.

Target rows:

- Zen4 `mlkem768/keygen`: 0.894x.
- Zen4 `mlkem1024/keygen`: 0.763x.
- Zen5 `mlkem768/keygen`: 0.902x.
- Zen5 `mlkem1024/keygen`: 0.779x.
- Ice Lake `mlkem768/keygen`: 0.859x.
- Ice Lake `mlkem1024/keygen`: 0.747x.
- Sapphire Rapids `mlkem768/keygen`: 0.930x.
- Sapphire Rapids `mlkem1024/keygen`: 0.811x.

Implementation scope:

1. Add an x86_64 SIMD rejection parser for public matrix samples:
   - AVX2 first, because it covers every x86_64 target in the loss set.
   - AVX512VL/BW optional after AVX2 if it materially helps Ice Lake and
     Sapphire Rapids.
2. Parse 24-byte or 48-byte chunks into candidate coefficients using SIMD
   unpack/shuffle/mask operations.
3. Keep compaction simple first:
   - Phase A: vector candidate extraction with scalar compact/store.
   - Phase B: vector mask-table compact/store only if Phase A is not enough.
4. Add x4 first-three-block matrix sampling for K=4 and K=3:
   - K=4: four lanes map naturally to one matrix row or row-major group.
   - K=3: three real lanes plus one dummy lane is allowed if it is faster than
     pair-plus-scalar.
5. Reuse existing `Shake128XofReader` x4 setup where possible. Do not mix a new
   Keccak implementation into this step unless measurement proves parsing is not
   the bottleneck.
6. Dispatch with `platform::caps()`:
   - AVX2 + SSE4.1 production path.
   - AVX512 extension only under explicit caps.
   - scalar fallback unchanged.

Correctness verification:

1. SIMD parser matches scalar parser for fixed seeds and all K=3/K=4 matrix
   coordinates.
2. Tests cover lane completion divergence, because rejection counts differ per
   lane.
3. Tests cover tail handling after the first three blocks.
4. `cargo miri` remains on portable fallback.
5. `cargo clippy` remains warning-free under the relevant feature set.

Performance verification:

1. Local x86_64 targeted benches show the parser row moving before production
   promotion.
   Verify:
   ```text
   cargo bench --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' \
     --bench auth \
     -- '^mlkem-sampler-phases/|^mlkem-keygen-matrix/k=[34]/|^(mlkem768|mlkem1024)/keygen/' \
     --quick
   ```
2. Inspect codegen for the parser.
   Verify: `cargo asm -p rscrypto <x86_sampler_function>` shows expected AVX2
   instructions and no accidental heap traffic or calls in the hot chunk loop.
3. Targeted Linux run covers Zen4, Zen5, Ice Lake, and Sapphire Rapids.
   Verify: all eight x86 target rows become ties or wins.

Promotion threshold:

- Promote AVX2 when it wins at least six of the eight x86 target rows and does
  not regress the remaining two by more than 1%.
- Add AVX512 only if Ice Lake/Sapphire Rapids remain losses after AVX2.

Rollback/rejection:

- If vector extraction wins diagnostics but scalar compaction caps the public
  win, keep Phase A behind `diag` and implement Phase B compact/store.
- If AVX2 parser does not move diagnostics, jump to Step 4 for x86 batched
  Keccak before writing assembly.

## Step 4: Batched Keccak Follow-Up

Objective: optimize SHAKE128 lane setup/squeezing only after parser work proves
insufficient. This prevents wasting effort on Keccak when rejection parsing is
still the dominant mismatch.

Entry criteria:

- Step 1 diagnostics show three-block parsing matters, but Step 2 or Step 3 does
  not fully close the public rows.
- Or Step 1 falsifies parser overhead and shows squeeze/setup dominates.

Implementation scope:

1. Split sampler diagnostics into:
   - XOF state setup,
   - first three squeezes,
   - three-block parse,
   - tail squeeze,
   - tail parse.
2. x86_64:
   - If AVX2 machines still lose, prototype AVX2 x4 Keccak for SHAKE128 matrix
     sampling.
   - Keep AVX512 x4 code paths separate and verify Zen4/Zen5 caps before
     assuming AVX512 helps.
3. AArch64 Linux:
   - Re-check SHA3 CE x3 hybrid and SVE2-SHA3 x4 dispatch.
   - Do not use single-state SHA3 CE on Graviton unless new data reverses the
     existing note that it is slower than portable.
4. Avoid general-purpose Keccak rewrites. The target is ML-KEM public matrix XOF
   shape: seeded 32-byte rho plus two public index bytes.

Correctness verification:

1. Hash/XOF test suite passes, not only ML-KEM.
   Verify: `cargo test --features 'sha3,shake,ml-kem,diag'`.
2. ML-KEM fixed vectors pass.
3. Cross-lane output ordering is tested directly against scalar `xof_seeded_32_2`.

Performance verification:

1. Keccak-specific rows improve:
   - `triple-xof-setup`
   - `triple-squeeze-3-blocks`
   - `quad-xof-setup`
   - `quad-squeeze-3-blocks`
2. Public keygen target rows improve after the lower-level rows improve.
3. Hash/XOF public benchmark rows do not regress, unless the new Keccak path is
   private to ML-KEM matrix sampling.

Promotion threshold:

- Promote only when Keccak changes close remaining public keygen losses or give
  at least 3% public keygen improvement on a still-losing platform.

Rejection criteria:

- If Keccak diagnostics improve but public keygen is flat, revert production
  dispatch and keep only useful diagnostics.

## Step 5: RISC-V And Power Campaigns

Objective: close the remaining non-x86/non-AArch64 losses after the shared
sampler work has landed or been rejected. These are separate architecture
campaigns, not tail work for the x86/AArch64 sampler patch.

### RISC-V

Target rows:

- RISE RISC-V `mlkem768/keygen`: 0.859x versus `aws-lc-rs`.
- RISE RISC-V `mlkem1024/keygen`: 0.812x versus `aws-lc-rs`.

Initial read:

- RISC-V loses in both matrix and NTT/noise phases.
- A first-three-block scalar parser may help, but it is unlikely to be enough by
  itself.
- The likely major path is RVV 1.0 for NTT, base multiplication, rejection
  sampling, or Keccak, depending on profiler evidence.

Work order:

1. Add RISC-V phase extraction to the target report.
   Verify: matrix delta, noise NTT delta, row sample, and tail rows are recorded
   for RISE RISC-V.
2. Capture CPU feature information from CI artifacts or runner logs.
   Verify: document whether RVV is available and which vector length is exposed.
3. Profile scalar RISC-V keygen before writing RVV.
   Verify: top symbols identify whether to start with NTT, rejection sampling,
   Keccak, or encode/from-product-domain.
4. If RVV is available, prototype the smallest useful RVV kernel:
   - first choice: rejection parser if sampler dominates,
   - second choice: forward NTT/product-domain conversion if NTT dominates,
   - third choice: cached basemul if matrix multiply dominates after sampling.
5. Keep scalar fallback unchanged.

Promotion threshold:

- RISC-V `mlkem768/keygen` and `mlkem1024/keygen` become ties or wins.
- If only one row closes, keep the backend behind architecture-specific dispatch
  only if it is clean, owned, and does not regress other RISC-V ML-KEM rows.

### Power10

Target row:

- Power10 `mlkem1024/keygen`: 0.940x versus `libcrux`.

Initial read:

- This is near tie threshold and should not distract from the twelve x86/AArch64
  losses.
- Because the gap is small, a shared sampler improvement may close it without a
  Power-specific backend.

Work order:

1. Re-check Power10 after Steps 1-4.
   Verify: if `mlkem1024/keygen` becomes >= 1.05x, classify as closed. If it
   moves only into the tie band, keep it on the follow-up list because the stated
   target is wins, not merely no losses.
2. If still below 0.95x, profile Power10 keygen phases.
   Verify: identify whether row sampling, NTT, or matrix multiply explains the
   remaining gap.
3. Prefer scalar cleanup or VSX intrinsics before assembly.
4. Consider Power10 assembly only if VSX intrinsics prove the algorithm and
   codegen leaves a clear gap.

Promotion threshold:

- Power10 `mlkem1024/keygen` becomes a tie or win.
- Power10 `mlkem768/keygen` remains a tie/win.

## 2026-07-02 x86_64 SIMD SampleNTT Parser Slice

Implemented the first production x86_64 parser slice instead of another
matrix-schedule experiment:

- Added an AVX2/SSE4.1/SSSE3 capability gate for public `SampleNTT` parsing.
- Added a `pshufb`-based 12-byte decoder that emits eight 12-bit candidates at
  a time in the same order as the scalar parser.
- Wired the parser into:
  - single-block public matrix sampling,
  - pair sampling,
  - quad materialized row sampling, which is the K=3/K=4 x86 keygen hot path,
  - the existing fused product fallback so it no longer keeps a separate scalar
    rejection loop.
- Kept bounded third-block handling explicit. The first two blocks use the
  full-capacity store path when the polynomial has room for all 112 possible
  candidates; the tail path caps writes at `N`.
- Added an x86-only scalar-equivalence test covering empty, partial, near-full,
  and full destinations.

Safety read:

- Public matrix rejection timing and write positions remain public-data
  dependent only.
- The SIMD loads are fixed 16-byte reads inside one 168-byte SHAKE128 rate
  block. The last 12-byte group uses a shifted load from offset 152 to avoid
  reading past the block.
- Secret polynomial arithmetic remains on the existing fixed-schedule AVX2
  multiply/NTT paths.

Local validation:

```text
cargo fmt --all -- --check
cargo check --features ml-kem
cargo check --target x86_64-unknown-linux-gnu --features ml-kem --lib
cargo clippy --features ml-kem --lib -- -D warnings
cargo check --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth
cargo test --features ml-kem sample_ntt_ -- --nocapture
cargo test --test mlkem_acvp --features ml-kem -- --nocapture --test-threads=1
cargo test --test mlkem_ops --features ml-kem -- --nocapture --test-threads=1
```

Local x86 unit-test execution/typecheck is blocked on this Apple Silicon host
by the missing `x86_64-linux-gnu-gcc` C/ASM toolchain pulled in by dev
dependencies. The production x86 library path compiles.

Targeted Linux quick bench:

```text
RUNNERS: zen4,zen5,icelake,sapphirerapids
PRIMITIVES/SELECTORS: mlkem768,mlkem1024
FILTERS: ^mlkem(768|1024)/keygen/rscrypto$|^mlkem-keygen-matrix/k=(3|4)/(materialized|fused|default)$|^mlkem-keygen-matrix/k=(3|4)/materialized-row-sample$|^mlkem-sampler-phases/k=4/(row-sample-counted|triple-parse-3-blocks|quad-parse-3-blocks)$
QUICK: true
```

Promotion rule:

- If materialized row sample and keygen both move on all four x86 Linux runners,
  keep this as the production parser and continue only if residual losses point
  to Keccak rather than rejection parsing.
- If row sample moves but keygen does not, profile matrix accumulate before
  changing schedule again.
- If row sample is flat or slower, delete or quarantine this parser; do not add
  another wrapper around it.

2026-07-02 targeted result:

Run
[`#28607261688`](https://github.com/loadingalias/rscrypto/actions/runs/28607261688)
validated commit `5cb1343610d3456501fd5a7c636c27ed872346a1` on the four x86_64
Linux runners with quick mode. The run measured rscrypto keygen plus the
targeted row-sampler diagnostics. It did not rerun external competitors, so
the fastest-external comparison below is against the frozen 2026-06-30 full
Linux baseline at commit `1bcc2d19cf7b2df93ad3e814a1763ab9ce7e84b4`.

| Platform | `mlkem768/keygen` before | After | Fastest external | Verdict | `mlkem1024/keygen` before | After | Fastest external | Verdict |
| --- | ---: | ---: | ---: | --- | ---: | ---: | ---: | --- |
| AMD Zen4 | 12.630 us | 10.274 us | 11.296 us | win | 20.724 us | 14.368 us | 15.821 us | win |
| AMD Zen5 | 9.259 us | 7.333 us | 8.355 us | win | 14.923 us | 10.283 us | 11.620 us | win |
| Intel Ice Lake | 16.046 us | 12.536 us | 13.778 us | win | 26.170 us | 17.930 us | 19.547 us | win |
| Intel Sapphire Rapids | 12.783 us | 11.470 us | 11.894 us | tie | 20.788 us | 16.360 us | 16.856 us | tie |

Decision:

- Promote and keep the owned x86_64 SIMD `SampleNTT` parser. It moved every
  targeted public keygen row materially: 11-28% faster for ML-KEM-768 and
  27-46% faster for ML-KEM-1024.
- The x86_64 target set moves from eight fastest-external losses to six wins
  and two conservative ties. Sapphire Rapids is still ahead of the frozen
  external rows by about 3-4%, but remains in the tie band under the 5%
  release threshold.
- No vendored or external-derived assembly was added or removed in this slice.
  The implementation is owned Rust/x86 intrinsics in the ML-KEM portable
  backend. `scripts/check/asm-ledger.sh` still reports `63 assembly files, 15
  owned, 48 external-derived`.
- The next full-scoreboard step is a full ML-KEM keygen comparison run that
  includes external competitors. Do not rewrite `benchmark_results/OVERVIEW.md`
  from this targeted rscrypto-only run.
- Remaining non-x86 keygen work is a separate campaign: Graviton3/4, RISC-V,
  and Power10 were not part of this x86 validation run.

## 2026-07-02 Non-X86 Matrix Dispatch Slice

Hypothesis:

- The next safe shared move is not more scheduler churn or a generic Keccak
  rewrite. The measured server-class paths that still mattered were either
  already on materialized matrix accumulation, or were still inheriting the
  generic fused fallback.
- RISC-V and Power10 artifacts already showed the analogous PKE matrix path
  prefers materialized accumulation over fused accumulation. That is enough
  evidence to stop using the generic fused keygen fallback on those lanes.
- Local AArch64 diagnostics showed the remaining non-Linux K=4 fused exception
  was stale: default K=4 tracked fused at roughly 13.4 us while materialized
  was roughly 11.0 us. Remove the exception instead of preserving a legacy
  branch.

Implemented change:

- AArch64, RISC-V, and PowerPC64 keygen matrix accumulation now use the owned
  materialized path by default.
- x86_64 keeps its existing AVX2/SSE4.1 materialized dispatch and scalar fused
  fallback.
- s390x remains materialized.
- The diagnostic bench now includes
  `mlkem-keygen-matrix/k={2,3,4}/default-through-matrix-accumulate`, so CI can
  validate the actual production dispatch against fused and materialized rows.

Local verification:

```text
cargo check --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth
cargo test --features 'ml-kem,diag' keygen_matrix --lib
cargo check --target aarch64-unknown-linux-gnu --features 'ml-kem,diag' --lib
cargo check --target riscv64gc-unknown-linux-gnu --features 'ml-kem,diag' --lib
cargo check --target powerpc64le-unknown-linux-gnu --features 'ml-kem,diag' --lib
cargo clippy --features 'ml-kem,diag' --lib -- -D warnings
cargo clippy --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth -- -D warnings
```

Local AArch64 quick bench after the dispatch cleanup:

```text
mlkem-keygen-matrix/k=4/default-through-matrix-accumulate      11.028 us
mlkem-keygen-matrix/k=4/materialized-through-matrix-accumulate 11.0 us band
```

Ownership result:

- No vendored assembly was added.
- No vendored assembly was removed.
- The change deliberately reuses owned Rust/ASM-backed materialized matrix
  machinery instead of adding another unproven target-specific kernel.

CI validation to run:

```text
RUNNERS: g3,g4,power10,riscv
  PRIMITIVES/SELECTORS: mlkem768,mlkem1024,mlkem_diag
  FILTERS: ^mlkem(768|1024)/keygen/(rscrypto|aws-lc-rs|libcrux)$|^mlkem-keygen-matrix/k=(3|4)/(default-through-matrix-accumulate|fused-through-matrix-accumulate|materialized-through-matrix-accumulate|windowed-through-matrix-accumulate|materialized-row-sample|materialized-row-multiply)$|^mlkem-keygen-phases/k=(3|4)/(through-noise-ntt|through-matrix-accumulate|through-encode)$
  QUICK: true
```

## 2026-07-02 AArch64 Direct Triple-XOF Sampler Slice

Run
[`#28616472237`](https://github.com/loadingalias/rscrypto/actions/runs/28616472237)
falsified the Linux AArch64 fused chunk-loop candidate. Public keygen stayed
flat while fused K=3/K=4 matrix rows regressed by about 1.6-2.2% on
Graviton3/4. The variable-count fused chunk-loop was deleted; do not revive
that direction without a new profile showing materialized is no longer the
dominant production path.

The next owned-code slice targets the materialized sampler directly:

- Added an AArch64 little-endian ML-KEM `Shake128` triple cursor that owns the
  three Keccak states and exposes rate blocks without constructing three
  generic XOF readers for production SampleNTT.
- Linux/macOS AArch64 `sample_ntt_triple_into` now parses main triple blocks
  directly from Keccak state memory and parses single-lane tails from state
  memory as well. The tail path reuses the existing owned one-block
  `rej_uniform` parser instead of materializing 168-byte stack buffers.
- The old reader-backed triple helpers compile only for non-direct targets,
  tests, or `diag`; production AArch64 no longer carries that legacy path.
- Added a cursor equivalence test proving the direct cursor emits the same
  SHAKE128 blocks as the existing three-reader path, including independent
  lane tails.

Safety read:

- This remains public matrix sampling. Rejection branches, fill counts, and
  write positions depend on public XOF bytes derived from public `rho`.
- The direct parser reads the first 168 bytes of a 25-lane Keccak state only on
  little-endian AArch64, where the in-memory `u64` layout matches SHAKE byte
  order.
- The cursor has a debug invariant that prevents returning to triple-block mode
  after any single-lane tail block has started.

Local validation:

```text
cargo check --features ml-kem,sha3
cargo check --features diag,ml-kem,sha3
cargo check --features ml-kem,sha3 --target aarch64-unknown-linux-gnu
cargo clippy --features ml-kem,sha3 -- -D warnings
cargo clippy --features diag,ml-kem,sha3 -- -D warnings
cargo clippy --features ml-kem,sha3 --target aarch64-unknown-linux-gnu -- -D warnings
cargo test --features ml-kem,sha3 --lib shake128_triple_cursor_matches_xof_readers
cargo test --features ml-kem,sha3 --lib sample_ntt_triple_matches_scalar_samplers
```

Targeted Linux quick bench:

```text
RUNNERS: g3,g4
PRIMITIVES/SELECTORS: mlkem768,mlkem1024,mlkem-keygen-matrix
FILTERS: ^mlkem(768|1024)/keygen/(rscrypto|aws-lc-rs)$|^mlkem-keygen-matrix/k=(3|4)/(default-through-matrix-accumulate|materialized-through-matrix-accumulate|fused-through-matrix-accumulate|windowed-through-matrix-accumulate|materialized-row-sample|materialized-row-multiply)$
QUICK: true
```

Promotion rule:

- Keep if `materialized-row-sample` and public keygen both move on Graviton3
  and Graviton4 without making fused/windowed look like the actual production
  answer.
- If row sample is flat, stop sampler wrapper work and profile Keccak/SampleNTT
  together before writing lower-level fused sampler assembly.
- If public keygen moves but row sample does not, broaden phase diagnostics
  before claiming this slice explains the gap.

Result:

Run
[`#28620265132`](https://github.com/loadingalias/rscrypto/actions/runs/28620265132)
validated commit `0bbce9cb8f3dfe79eaafe04facc35322b1c2b345` on Graviton3 and
Graviton4. The direct triple-XOF cursor failed the promotion rule.

| Platform | Row | rscrypto | AWS-LC | Ratio | Read |
| --- | --- | ---: | ---: | ---: | --- |
| Graviton3 | `mlkem768/keygen` | 17.008 us | 13.916 us | 0.818x | flat/loss |
| Graviton3 | `mlkem1024/keygen` | 26.932 us | 19.819 us | 0.736x | flat/loss |
| Graviton4 | `mlkem768/keygen` | 14.990 us | 11.584 us | 0.773x | flat/loss |
| Graviton4 | `mlkem1024/keygen` | 23.438 us | 16.352 us | 0.698x | flat/loss |

The sampler diagnostics did not improve:

| Platform | Row | Result | Previous same-day materialized row | Delta |
| --- | --- | ---: | ---: | ---: |
| Graviton3 | K=3 `materialized-row-sample` | 7.0401 us | 7.0189 us | 0.30% slower |
| Graviton3 | K=4 `materialized-row-sample` | 13.239 us | 13.185 us | 0.41% slower |
| Graviton4 | K=3 `materialized-row-sample` | 6.2859 us | 6.2746 us | 0.18% slower |
| Graviton4 | K=4 `materialized-row-sample` | 11.572 us | 11.545 us | 0.23% slower |

Decision:

- Reject and revert the direct triple-XOF cursor production path. It adds
  backend surface area without measurable Graviton benefit.
- Stop writing Rust-level wrappers around the same Keccak/rejection boundary.
  Three-block parsing, generic Keccak x3, fused chunk-loop multiply, and direct
  XOF cursor attempts have all failed on Graviton.
- The next credible implementation must either be backed by a Graviton profile
  that identifies a different dominant phase, or be a genuinely fused
  ML-KEM-specific matrix sampler kernel that owns Keccak block generation and
  rejection parsing together. Do not spend another paid Graviton run on
  sampler scheduling or reader-structure changes.

## 2026-07-03 Graviton Profiling Harness

The first direct profiler attempt used the full `auth` Criterion bench on the
G3/G4 dev machines. That was the wrong profiler target. It pulled in Criterion,
external oracle crates, and target-specific `dev-dependencies`, then spent the
remote session on first-run build/link work. Both machines later lost SSH
reachability over Tailscale relay before useful `perf` artifacts were
retrieved. The boxes were stopped cleanly.

The replacement is a real package binary, not an example and not a Criterion
bench:

```text
cargo build --profile bench --features ml-kem,diag --bin mlkem_profile
target/release/mlkem_profile --list
target/release/mlkem_profile --case keygen1024 --seconds 20
```

Why it is a binary:

- `src/bin/mlkem_profile.rs` depends only on normal `rscrypto` dependencies.
- `examples/` are dev targets; they can still pull target-specific
  `dev-dependencies` such as AWS-LC/libcrux support crates into a cross-target
  graph.
- `profile.bench` keeps release optimization with debug info for useful
  `perf report` symbols.

Remote profiler entry point:

```text
scripts/bench/mlkem-profile-remote.sh g3
scripts/bench/mlkem-profile-remote.sh g4
```

The helper starts and syncs one host, builds only `mlkem_profile`, launches a
detached `perf` run, retrieves artifacts into ignored `benchmark_results/`, and
stops the instance by default. It disables the dev-machine idle-shutdown timer
for the run because detached `perf` has no live SSH session and otherwise looks
idle to the five-minute detector. `perf stat` is skipped by default because
`perf stat -d` segfaulted on Graviton3 while reading derived slots metrics; the
authoritative artifact is `perf record` plus `perf report`. The helper also
normalizes ownership on `sudo perf record` outputs before retrieval. Default
cases are intentionally narrow:

```text
keygen768 keygen1024
k3-matrix-default k4-matrix-default
k3-row-sample k4-row-sample
k3-row-multiply k4-row-multiply
k4-sampler-squeeze3 k4-sampler-parse3
```

Promotion rule: do not write another AArch64 ML-KEM implementation slice until
the G3/G4 `perf.report.txt` files identify whether the remaining time is in
Keccak/XOF, rejection parsing, matrix multiply/accumulate, noise NTT, or
encoding/from-product-domain exit.

## 2026-07-03 Graviton `perf` Result

Artifacts:

```text
benchmark_results/profiles/mlkem/linux-g3-20260703T041249Z/
benchmark_results/profiles/mlkem/linux-g4-20260703T041658Z/
```

The profile result is decisive: the remaining Graviton keygen gap is dominated
by Keccak/XOF inside public matrix sampling. The owned rejection parser and the
cached row multiply path are visible, but they are not the largest remaining
buckets.

Representative K=4 samples:

| Host | Row | Matrix/sample bucket | Keccak/XOF bucket | Parser bucket | Row multiply bucket |
| --- | --- | ---: | ---: | ---: | ---: |
| G3 | `keygen1024` | 55.99% matrix | 39.41% `keccakf_sha3_x3_hybrid_impl` | 6.12% triple parser | 7.43% cached multiply |
| G4 | `keygen1024` | 56.68% matrix | 42.55% `keccakf_sha3_x3_hybrid_impl` | 5.72% triple parser | 6.58% cached multiply |
| G3 | `k4-matrix-default` | 71.02% matrix | 49.95% `keccakf_sha3_x3_hybrid_impl` | 8.15% triple parser | 9.40% cached multiply |
| G4 | `k4-matrix-default` | 71.42% matrix | 52.78% `keccakf_sha3_x3_hybrid_impl` | 7.19% triple parser | 8.52% cached multiply |
| G3 | `k4-row-sample` | 71.90% `sample_ntt_triple_into` | 69.48% `keccakf_sha3_x3_hybrid_impl` | 12.93% triple parser | n/a |
| G4 | `k4-row-sample` | 75.37% `sample_ntt_triple_into` | 72.44% `keccakf_sha3_x3_hybrid_impl` | 11.67% triple parser | n/a |

The split rows confirm the read:

- `k4-sampler-squeeze3` is almost entirely
  `keccakf_sha3_x3_hybrid_impl`.
- `k4-sampler-parse3` spends about 26-27% in the owned triple parser, but that
  parser is only about 5-8% of keygen/matrix once Keccak is included.
- `k4-row-multiply` shows the cached NEON multiply kernel at about 18-20% of a
  multiply-only diagnostic loop, but only about 6-9% inside full keygen/matrix.
- The end-to-end `h` hash is a secondary K=4 keygen bucket at about 15%, after
  the public matrix sampler.

Decision:

1. Stop parser-first AArch64 work. Triple-block parser assembly was worth
   owning, but the next parser change cannot close the gap by itself.
2. Stop row-multiply and fused matrix scheduling work until Keccak/XOF changes.
   Those paths are not the dominant remaining Graviton cost.
3. The next implementation must target the public matrix XOF directly:
   `sample_ntt_triple_into` and `keccakf_sha3_x3_hybrid_impl`, preferably with
   an owned ML-KEM-specific AArch64 SHAKE128/Keccak path that keeps the
   three-lane matrix-sampling workload in a shape the parser can consume without
   reader/scratch churn.
4. Constant-time boundary: this target is acceptable because matrix sampling
   uses `rho` and public matrix coordinates. Rejection loop counts and accepted
   candidates depend on public XOF bytes, not on secret `s`, `e`, or `sigma`.

The concrete next slice is not another benchmark run. Inspect the current
AArch64 Keccak backend and implement an owned matrix-sampler XOF kernel, then
validate with the same `mlkem_profile` rows before spending GitHub Actions
bench time.

## 2026-07-03 Matrix-XOF Implementation Slice

Implemented the first production AArch64 matrix-XOF change:

- Added an ML-KEM matrix-specific SHAKE128 triple reader on Linux AArch64.
- The matrix reader initializes the three public-matrix XOF states with a
  split schedule: SHA3 CE `x2` for the first two states plus the tuned scalar
  permutation for the third state.
- Later rate-block callbacks use the same split schedule instead of the generic
  `keccakf_sha3_x3_hybrid_impl` path.
- Production `sample_ntt_triple_into` and the sampler diagnostics now route
  through the matrix reader on Linux AArch64.
- The rejection parser, row multiply, fused matrix paths, and NTT paths were not
  changed.

Why this is the right first cut:

- The generated AArch64 assembly for `keccakf_sha3_x3_hybrid_impl` reserves a
  336-byte stack frame and spills scalar/vector state through the round loop.
- The optimized `sample_ntt_triple_into` path now calls
  `keccakf_sha3_x2_impl` plus `keccakf_portable`; it no longer calls
  `keccakf_sha3_x3_hybrid_impl`.
- This directly attacks the profiled bucket while leaving the already-owned
  parser and multiply kernels alone.

Security boundary:

- The changed path handles only public matrix-A sampling from `rho` and public
  matrix coordinates.
- Rejection counts and rate-block loop counts remain public-data-dependent.
- Secret noise sampling, secret NTTs, and secret-dependent arithmetic dispatch
  are unchanged.

Validation before promotion:

```text
cargo fmt --all -- --check
cargo check --features ml-kem,diag --lib
cargo check --features ml-kem,diag --lib --target aarch64-unknown-linux-gnu
cargo clippy --features ml-kem,diag --lib -- -D warnings
cargo clippy --features ml-kem,diag --lib --target aarch64-unknown-linux-gnu -- -D warnings
cargo test --features ml-kem,diag --lib sample_ntt_triple_matches_scalar_samplers -- --nocapture
cargo test --features ml-kem,diag --lib materialized_k4_matrix_accumulate_matches_reference_layouts -- --nocapture
cargo test --features ml-kem,diag --lib materialized_k3_matrix_accumulate_matches_reference_layouts -- --nocapture
cargo check --features ml-kem,diag --bin mlkem_profile --target aarch64-unknown-linux-gnu
```

Remote validation should be exact and quick:

```text
RUNNERS: g3,g4
CASES: keygen768 keygen1024 k3-matrix-default k4-matrix-default k3-row-sample k4-row-sample k4-sampler-squeeze3
MODE: quick/profile first, then GitHub quick bench only if the profile rows move
```

2026-07-03 quick result:

Run
[`#28666127325`](https://github.com/loadingalias/rscrypto/actions/runs/28666127325)
validated commit `7cefb0d50464b392fdc9d312316026c83de75e9f` on Graviton3 and
Graviton4. The split `x2 + scalar` matrix-XOF production path must be rejected.

| Platform | Row | rscrypto | AWS-LC | Ratio | Previous rscrypto | Delta |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| Graviton3 | `mlkem768/keygen` | 19.871 us | 13.889 us | 0.699x | 17.008 us | 16.8% slower |
| Graviton3 | `mlkem1024/keygen` | 31.726 us | 19.786 us | 0.624x | 26.932 us | 17.8% slower |
| Graviton4 | `mlkem768/keygen` | 17.324 us | 11.564 us | 0.667x | 14.990 us | 15.6% slower |
| Graviton4 | `mlkem1024/keygen` | 27.396 us | 16.300 us | 0.595x | 23.438 us | 16.9% slower |

Sampler diagnostics moved the same direction:

| Platform | Row | Result | Previous materialized row | Delta |
| --- | --- | ---: | ---: | ---: |
| Graviton3 | K=3 `materialized-row-sample` | 9.9154 us | 7.0189 us | 41.3% slower |
| Graviton3 | K=4 `materialized-row-sample` | 17.999 us | 13.185 us | 36.5% slower |
| Graviton4 | K=3 `materialized-row-sample` | 8.7265 us | 6.2746 us | 39.1% slower |
| Graviton4 | K=4 `materialized-row-sample` | 15.591 us | 11.545 us | 35.0% slower |

Decision: revert the split `x2 + scalar` matrix-XOF route. It removed the
hybrid call from the production sampler, but it destroyed throughput because the
third scalar permutation no longer overlaps the SHA3 CE pair. The next viable
path is not another Rust reader reshape; it is either a real owned matrix-XOF
kernel that keeps the third state overlapped without the compiler's 336-byte
hybrid spill pattern, or a targeted rollback followed by a lower-level
hand-written AArch64 kernel.

## End-To-End Verification Gate

Every promoted candidate must complete this gate before being considered closed:

1. Formatting and lint:
   ```text
   cargo fmt --all -- --check
   just check
   ```
2. ML-KEM correctness:
   ```text
   cargo test --features 'ml-kem,diag' auth::mlkem::portable::tests:: -- --nocapture
   cargo test --test mlkem_acvp --features ml-kem -- --nocapture
   cargo test --test mlkem_ops --features ml-kem -- --nocapture
   ```
3. Portable safety:
   ```text
   just test-miri
   ```
4. Targeted local benchmark:
   ```text
   cargo bench --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' \
     --bench auth \
     -- '^mlkem-sampler-phases/|^mlkem-keygen-matrix/|^mlkem-matrix-sample/|^(mlkem512|mlkem768|mlkem1024)/keygen/' \
     --quick
   ```
5. Targeted Linux CI benchmark:
   ```text
   PRIMITIVES/SELECTORS: mlkem512,mlkem768,mlkem1024
   FILTERS: ^mlkem-sampler-phases/|^mlkem-keygen-matrix/|^mlkem-matrix-sample/|^(mlkem512|mlkem768|mlkem1024)/keygen/
   ```
6. Full Linux benchmark confirmation after target rows close.
   Verify: `benchmark_results/OVERVIEW.md` shows both target primitive/op rows
   with no losses.

## Documentation And Ledger Updates

Each promoted implementation must update:

1. `docs/tasks/mlkem-ntt-notebook.md` with the measured result and whether the
   old hypothesis was confirmed or rejected.
2. `docs/tasks/asm.md` for any new assembly file, including ownership status.
3. `benchmark_results/OVERVIEW.md` after each extracted CI run.
4. This plan with:
   - run id,
   - commit,
   - changed files,
   - target-row before/after table,
   - promotion or rejection decision.

## Stop Conditions

Stop and reassess if any of these happen:

- A candidate wins diagnostics but repeatedly loses public keygen.
- The implementation needs broad end-to-end keygen assembly to explain a narrow
  sampler gap.
- A candidate closes `mlkem1024/keygen` by breaking `mlkem768/keygen` or
  `mlkem512/keygen`.
- x86 and AArch64 results diverge enough that one shared abstraction starts
  forcing unnatural code on either architecture.
- New code introduces ownership ambiguity under `docs/tasks/asm.md`.

The default response to a failed candidate is deletion or `diag` quarantine, not
another layer of dispatch.
