# ML-KEM Forward NTT Ownership Notebook

Started: 2026-06-26

Goal: remove the mlkem-native / NeonNTT / SLOTHY-derived AArch64 forward NTT
assembly and keep ML-KEM keygen on an owned, constant-time backend. The trigger
was the post-P-384 `benchmark_results/OVERVIEW.md` view: `mlkem1024/keygen`
was the worst remaining ASM-linked top-loss area, with `mlkem768/keygen` also
in the top loss set.

This is an ownership slice first. If Graviton benches show the owned NEON path
does not close the keygen gap, the next step is an rscrypto-generated AArch64
kernel from the FIPS 203 schedule, not restoring the vendored files.

## Starting Baseline

Benchmark source: `benchmark_results/OVERVIEW.md`, Linux CI run
`#28202726811`, created 2026-06-25 21:50:27 UTC, commit
`c5ece53efd98980831de728974d1f588cf055ca5`.

Relevant public rows:

| Group | Rows | W/T/L | Geomean | Median | Pressure |
| --- | ---: | ---: | ---: | ---: | --- |
| `mlkem1024` / `keygen` | 9 | 3/2/4 | 0.95x | 1.03x | `aws-lc-rs` 3, `libcrux` 1 |
| `mlkem768` / `keygen` | 9 | 4/2/3 | 0.98x | 1.02x | `aws-lc-rs` 3 |

Worst ASM-linked rows:

- Graviton4 `mlkem1024/keygen`: 0.683x versus AWS-LC.
- Graviton3 `mlkem1024/keygen`: 0.713x versus AWS-LC.
- Graviton4 `mlkem768/keygen`: 0.741x versus AWS-LC.
- Graviton3 `mlkem768/keygen`: 0.762x versus AWS-LC.

RISC-V and POWER10 losses in the same primitive family are not caused by the
AArch64 forward NTT assembly. They remain separate non-ASM or non-AArch64
performance work.

## Deleted Files

External-derived forward NTT:

- `src/auth/mlkem/asm/rscrypto_mlkem_ntt_aarch64_apple_darwin.s`
- `src/auth/mlkem/asm/rscrypto_mlkem_ntt_aarch64_linux.s`

These files carried an rscrypto-owned canonicalization epilogue, but their
butterfly schedule was derived from mlkem-native / NeonNTT / SLOTHY. The live
AArch64 forward NTT now routes through the existing rscrypto-owned Rust NEON
kernel in `src/auth/mlkem/portable.rs`.

No-longer-used owned inverse NTT:

- `src/auth/mlkem/asm/rscrypto_mlkem_inv_ntt_aarch64_apple_darwin.s`
- `src/auth/mlkem/asm/rscrypto_mlkem_inv_ntt_aarch64_linux.s`

These were rscrypto-owned, not vendored. During verification, macOS
encapsulation and decapsulation exposed the Darwin inverse assembly as unsafe
to keep even as a test-only candidate. Linux production already used the Rust
NEON inverse path, so the safe cleanup was to route macOS inverse NTT through
the same owned NEON path and delete the unused inverse assembly pair.

## Constant-Time Boundary

The promoted NEON kernel uses fixed public ML-KEM dimensions and public zeta
indices. Its loops and memory addresses do not depend on secret polynomial
coefficients. Modular arithmetic uses NEON multiply, add, subtract, compare,
and mask operations; it does not add division or coefficient-dependent branch
control to the secret path.

The public `SampleNTT` rejection parser assembly and owned base-multiply /
inverse-NTT assembly are intentionally out of scope. This slice removes only
the external-derived forward NTT files.

## Implementation Notes

- Production AArch64 `ntt(poly)` now calls `ntt_neon(poly)` on macOS and Linux
  instead of the deleted assembly symbols.
- The `diag` surface was renamed from `aarch64_ntt_asm` to
  `aarch64_ntt_neon` so benchmark output does not lie about the backend.
- The ML-KEM AArch64 gate now runs the `ntt_neon` scalar-oracle tests.
- Production AArch64 inverse NTT and inverse NTT plus add now use the owned
  Rust NEON path on both macOS and Linux.
- The assembly ledger dropped from 62 to 58 tracked `.s` / `.S` files, from 49
  to 47 external-derived files, and from 13 to 11 owned assembly files.

## Verification

Local AArch64 macOS verification:

```bash
cargo test --lib --features ml-kem ntt_neon -- --nocapture
```

Result: 4 tests passed. Coverage includes canonical output, 1024 seeded
polynomials against scalar NTT, fixed FIPS keygen-noise inputs, and forward /
inverse round-trip checks for the NEON path.

Required before declaring the performance side closed:

```text
PRIMITIVES/SELECTORS: mlkem512,mlkem768,mlkem1024
RUNNERS: g3,g4
FILTERS: keygen
```

Use `g3,g4` first because the triggering ASM-linked losses are Graviton rows.
If those win or tie while x86 and non-AArch64 behavior remains unchanged, this
slice is closed. If they regress, keep the deletion direction but replace the
Rust NEON path with an rscrypto-generated AArch64 forward NTT kernel.

## 2026-06-26 Follow-up AArch64 Keygen Tuning

Run
[`#28250494646`](https://github.com/loadingalias/rscrypto/actions/runs/28250494646)
kept the forward-NTT deletion correct but did not close the Graviton keygen
gap. The worst remaining rows were still G3/G4 keygen against AWS-LC, with
ML-KEM-1024 around 0.62x-0.64x and ML-KEM-512/768 around 0.65x-0.67x.

Kept candidate:

- AArch64 forward NTT now uses lazy signed NEON butterflies and one final
  canonicalization pass, avoiding per-butterfly modular add/sub reductions.
- Keygen samples `e` directly into `t_hat` and runs a fused
  NTT-to-product-domain finalizer, removing one `PolyVec` stack allocation and
  one copy.
- AArch64 keygen enables the existing fused matrix accumulation only for K=4.
  Local K=2/K=3 fused paths remained slower, so those stay materialized.
- Added direct NEON tests for the fused product-domain NTT and for the lazy
  finalizer over the documented `[-8Q, 8Q]` range.

Rejected during this pass: deleting the owned AArch64 SampleNTT rejection
parser assembly. The Rust/NEON fallback is correct, but local diagnostics made
the fused matrix path materially slower, with K=4 fused matrix timing roughly
doubling. Keep the compact parser until an owned replacement beats it; it is
already rscrypto-owned and is not part of the external-derived forward-NTT
bucket.

Local Apple Silicon production-keygen signal after the kept candidate:

| Row | rscrypto | AWS-LC |
| --- | ---: | ---: |
| `mlkem512/keygen` | 5.115 us | 6.366 us |
| `mlkem768/keygen` | 8.298 us | 9.228 us |
| `mlkem1024/keygen` | 11.990 us | 12.846 us |

Verification:

```bash
cargo fmt --all --check
cargo check --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth
cargo test --lib --features ml-kem ntt -- --nocapture
cargo test --lib --features ml-kem lazy_ntt_neon_finalizers_match_scalar_range -- --nocapture
cargo test --test mlkem_acvp --features ml-kem -- --nocapture
cargo test --test mlkem_ops --features ml-kem -- --nocapture
cargo bench --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem' --bench auth -- \
  "^(mlkem512|mlkem768|mlkem1024)/keygen/(rscrypto|aws-lc-rs)$" --quick
```

Required Graviton check before closing the performance side:

```text
PRIMITIVES/SELECTORS: mlkem512,mlkem768,mlkem1024
RUNNERS: g3,g4
FILTERS: keygen
QUICK: true
```

## 2026-06-27 Keygen Phase Scoreboard

Run
[`#28276488752`](https://github.com/loadingalias/rscrypto/actions/runs/28276488752)
made the next target clear. The public keygen losses are no longer just an
AArch64 forward-NTT ownership problem:

| Platform | Row | rscrypto | Fastest external | Ratio |
| --- | --- | ---: | ---: | ---: |
| AWS Graviton4 | `mlkem1024/keygen` vs AWS-LC | 23.406 us | 16.333 us | 0.70x |
| AWS Graviton3 | `mlkem1024/keygen` vs AWS-LC | 26.857 us | 19.860 us | 0.74x |
| Intel Ice Lake | `mlkem1024/keygen` vs `libcrux` | 25.981 us | 19.506 us | 0.75x |
| AMD Zen4 | `mlkem1024/keygen` vs `libcrux` | 21.059 us | 15.848 us | 0.75x |
| AWS Graviton4 | `mlkem768/keygen` vs AWS-LC | 14.884 us | 11.601 us | 0.78x |

The `mlkem-keygen-phases` rows are diagnostic rscrypto-only timings. The
`expand-seed` row measures `G(d || k)` separately. The remaining rows are
cumulative from already-expanded `rho` and `sigma`, so the useful deltas are:

- `noise-ntt = through-noise-ntt - noise-sample`
- `matrix-accumulate = through-matrix-accumulate - through-noise-ntt`
- `tail = through-encode - through-matrix-accumulate`

Representative medians from the same artifact set:

| Platform | K | Diag total | Noise sample | Noise NTT | Matrix accumulate | Tail | Matrix share |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| AWS Graviton3 | 2 | 8.894 us | 3.111 us | 1.466 us | 3.994 us | 0.323 us | 45% |
| AWS Graviton3 | 3 | 13.173 us | 2.475 us | 2.182 us | 7.986 us | 0.529 us | 61% |
| AWS Graviton3 | 4 | 21.472 us | 3.127 us | 2.914 us | 14.790 us | 0.642 us | 69% |
| AWS Graviton4 | 2 | 7.524 us | 2.559 us | 1.195 us | 3.450 us | 0.320 us | 46% |
| AWS Graviton4 | 3 | 11.491 us | 2.168 us | 1.821 us | 7.074 us | 0.428 us | 62% |
| AWS Graviton4 | 4 | 18.607 us | 2.712 us | 2.414 us | 12.892 us | 0.589 us | 69% |
| AMD Zen4 | 4 | 16.562 us | 1.278 us | 1.394 us | 13.392 us | 0.497 us | 81% |
| AMD Zen5 | 4 | 11.641 us | 0.931 us | 1.027 us | 9.442 us | 0.241 us | 81% |
| Intel Ice Lake | 4 | 20.108 us | 1.731 us | 1.997 us | 15.558 us | 0.822 us | 77% |
| Intel Sapphire Rapids | 4 | 15.936 us | 1.470 us | 1.652 us | 12.741 us | 0.073 us | 80% |
| RISE RISC-V | 4 | 338.680 us | 25.546 us | 119.424 us | 181.500 us | 12.210 us | 54% |

Conclusion: do not write owned ASM for keygen end to end. The dominant
diagnostic cost is `keygen_matrix_accumulate`, especially for K=3 and K=4.
Forward NTT is not large enough to explain the current top losses by itself.

The next split is inside matrix accumulation:

| Platform | K | Matrix phase delta | Best matrix-sample row | Sample time | Best multiply-accumulate row | Multiply time |
| --- | ---: | ---: | --- | ---: | --- | ---: |
| AWS Graviton3 | 4 | 14.790 us | `triple` | 13.060 us | default | 3.312 us |
| AWS Graviton4 | 4 | 12.892 us | `triple` | 11.664 us | `asm-owned` | 3.073 us |
| AMD Zen4 | 4 | 13.392 us | `quad` | 7.598 us | default | 1.518 us |
| AMD Zen5 | 4 | 9.442 us | `quad` | 5.151 us | default | 0.861 us |
| Intel Ice Lake | 4 | 15.558 us | `quad` | 7.774 us | default | 2.177 us |
| Intel Sapphire Rapids | 4 | 12.741 us | `quad` | 6.369 us | default | 1.905 us |
| RISE RISC-V | 4 | 181.500 us | `pair` | 101.570 us | default | 28.183 us |

Next work item:

1. Isolate `keygen_matrix_accumulate::<K>` in `src/auth/mlkem/portable.rs`.
2. Split its benchmark into SampleNTT/rejection parsing, pointwise
   multiply-accumulate, and store/finalization costs.
3. Start with K=4 because it drives the worst `mlkem1024/keygen` losses.
4. On AArch64, keep the existing owned multiply-accumulate assembly unless a
   local replacement wins. The data points first at matrix sampling/rejection
   throughput, not at end-to-end keygen assembly.
5. After a candidate change, rerun:

```text
RUNNERS: g3,g4,amd-zen4,amd-zen5,intel-icl,intel-spr
FILTERS: ^mlkem-keygen-phases/|^mlkem-matrix-sample/|^mlkem-arithmetic/|^(mlkem512|mlkem768|mlkem1024)/keygen/
QUICK: true
```

## 2026-06-27 Matrix Split Bench Additions

Added a `diag`-only `mlkem-keygen-matrix` benchmark group. The new rows force
the two keygen matrix strategies and add a materialized row-multiply row:

- `mlkem-keygen-matrix/k=2/fused-through-matrix-accumulate`
- `mlkem-keygen-matrix/k=2/materialized-through-matrix-accumulate`
- `mlkem-keygen-matrix/k=2/materialized-row-multiply`
- `mlkem-keygen-matrix/k=3/fused-through-matrix-accumulate`
- `mlkem-keygen-matrix/k=3/materialized-through-matrix-accumulate`
- `mlkem-keygen-matrix/k=3/materialized-row-multiply`
- `mlkem-keygen-matrix/k=4/fused-through-matrix-accumulate`
- `mlkem-keygen-matrix/k=4/materialized-through-matrix-accumulate`
- `mlkem-keygen-matrix/k=4/materialized-row-multiply`

The production strategy branch was factored into
`keygen_matrix_accumulate_fused` and
`keygen_matrix_accumulate_materialized`; production selection still goes
through `use_fused_matrix_accumulate`.

Local Apple Silicon quick signal:

| Row | Median |
| --- | ---: |
| `k=2/fused-through-matrix-accumulate` | 3.7545 us |
| `k=2/materialized-through-matrix-accumulate` | 3.6315 us |
| `k=2/materialized-row-multiply` | 2.8477 us |
| `k=3/fused-through-matrix-accumulate` | 6.9591 us |
| `k=3/materialized-through-matrix-accumulate` | 6.3464 us |
| `k=3/materialized-row-multiply` | 5.4337 us |
| `k=4/fused-through-matrix-accumulate` | 9.3015 us |
| `k=4/materialized-through-matrix-accumulate` | 10.578 us |
| `k=4/materialized-row-multiply` | 8.8627 us |

Interpretation: K=2/K=3 should stay materialized locally. K=4 still wants
forced-strategy confirmation on Linux AArch64 because local Apple Silicon says
fused is faster, but the current Graviton production gate uses materialized.
Do not flip the Linux gate from local data alone.

Next run:

```text
RUNNERS: g3,g4
FILTERS: ^mlkem-keygen-matrix/|^mlkem-keygen-phases/|^mlkem-matrix-sample/|^(mlkem512|mlkem768|mlkem1024)/keygen/
QUICK: true
```

Decision rule:

- If Graviton K=4 `fused-through-matrix-accumulate` beats
  `materialized-through-matrix-accumulate`, flip the AArch64 Linux K=4 gate and
  rerun public `mlkem1024/keygen`.
- If materialized still wins on Graviton, keep the gate and attack
  materialized row multiply first. The local row-multiply cost is already most
  of K=4 materialized matrix time.

## 2026-06-27 Graviton Matrix Decision

Run
[`#28297025116`](https://github.com/loadingalias/rscrypto/actions/runs/28297025116)
was a targeted G3/G4 quick run at commit
`9843d8645c73eb374f4aa15a14e700abc883e700`.

The artifacts were parsed from `/tmp/rscrypto-bench-28297025116` instead of
being copied into `benchmark_results/2026-06-27/linux/`, because that date
already contains the broader Linux overview run.

Forced matrix strategy medians:

| Platform | K | Fused through matrix | Materialized through matrix | Faster | Materialized row multiply |
| --- | ---: | ---: | ---: | --- | ---: |
| AWS Graviton3 | 2 | 9.609 us | 8.570 us | materialized by 1.12x | 3.023 us |
| AWS Graviton3 | 3 | 16.964 us | 12.642 us | materialized by 1.34x | 5.769 us |
| AWS Graviton3 | 4 | 22.586 us | 20.807 us | materialized by 1.09x | 9.324 us |
| AWS Graviton4 | 2 | 8.126 us | 7.203 us | materialized by 1.13x | 2.791 us |
| AWS Graviton4 | 3 | 14.238 us | 11.052 us | materialized by 1.29x | 5.315 us |
| AWS Graviton4 | 4 | 19.228 us | 17.997 us | materialized by 1.07x | 8.596 us |

Current public keygen signal from the same run:

| Platform | Row | rscrypto | AWS-LC | Ratio |
| --- | --- | ---: | ---: | ---: |
| AWS Graviton3 | `mlkem512/keygen` | 11.512 us | 8.564 us | 0.74x |
| AWS Graviton3 | `mlkem768/keygen` | 16.912 us | 13.930 us | 0.82x |
| AWS Graviton3 | `mlkem1024/keygen` | 26.926 us | 19.943 us | 0.74x |
| AWS Graviton4 | `mlkem512/keygen` | 9.880 us | 7.167 us | 0.73x |
| AWS Graviton4 | `mlkem768/keygen` | 14.900 us | 11.584 us | 0.78x |
| AWS Graviton4 | `mlkem1024/keygen` | 23.444 us | 16.337 us | 0.70x |

Decision: keep the Linux AArch64 materialized matrix gate. The forced fused
path loses on both G3 and G4 for K=2, K=3, and K=4. Do not flip the K=4 Linux
gate.

Next work item: attack materialized row multiply and row layout. For K=4,
row-multiply alone is 9.324 us on G3 and 8.596 us on G4, which is roughly
45-48% of forced materialized through-matrix time. The next diagnostic should
separate cached row multiplication from cache construction and row storage so
we know whether the waste is in `polyvec_mulcache`, four row dot-products, or
materialized row layout.

## 2026-06-27 K=4 Materialized Split

Added K=4-only materialized split rows to `mlkem-keygen-matrix`:

- `k=4/materialized-row-sample`
- `k=4/materialized-row-multiply-default-input`
- `k=4/materialized-mulcache-input`
- `k=4/materialized-row-multiply-cached-input`
- `k=4/materialized-row-multiply-cached-core-input`

Local AArch64 quick signal:

| Row | Median |
| --- | ---: |
| `k=4/materialized-through-matrix-accumulate` | 10.541 us |
| `k=4/materialized-row-multiply` | 8.706 us |
| `k=4/materialized-row-sample` | 7.052 us |
| `k=4/materialized-row-multiply-default-input` | 1.552 us |
| `k=4/materialized-mulcache-input` | 0.169 us |
| `k=4/materialized-row-multiply-cached-input` | 1.314 us |
| `k=4/materialized-row-multiply-cached-core-input` | 1.141 us |

Correction: the older `materialized-row-multiply` diagnostic is polluted by
deterministic input construction. It is not a clean row-multiply cost. The
input-based split says the AArch64 cached path is already better than the
default dot-product path on the same fixed inputs:

- cached core: 1.141 us
- cached including `polyvec_mulcache`: 1.314 us
- default dot product: 1.552 us

Conclusion: do not attack the cached row multiply first. The row dot-products
and RHS mulcache are too small to explain the ML-KEM-1024 keygen loss. The
dominant local cost is materialized row sampling/rejection parsing and row
layout: `materialized-row-sample` is 7.052 us before any cached dot-products.

Next G3/G4 run:

```text
RUNNERS: g3,g4
FILTERS: ^mlkem-keygen-matrix/k=4/materialized-|^(mlkem1024)/keygen/
QUICK: true
```

Decision rule:

- If G3/G4 matches local shape, stop trying to win keygen with owned
  row-multiply ASM. The next implementation target is the K=4 materialized
  sampler: fewer SHAKE squeezes, tighter rejection parsing, or direct
  sample-and-pack into the row layout consumed by cached multiplication.
- If G3/G4 says cached core or mulcache is unexpectedly large, revisit the
  AArch64 cached NEON kernel and compare it against the owned ASM dot-product
  path with the same fixed inputs.

## 2026-06-27 G3/G4 K=4 Materialized Split Decision

Run
[`#28298168464`](https://github.com/loadingalias/rscrypto/actions/runs/28298168464)
was a targeted G3/G4 quick run at commit
`2f103b97a63decff488d762d36aa92fe43182037`.

Artifacts were parsed from `/tmp/rscrypto-bench-28298168464` and not copied
into `benchmark_results/2026-06-27/linux/`, because the tracked date directory
already holds broader Linux data and this run's artifact headers use per-job
times instead of the run creation time.

Targeted medians:

| Platform | Public keygen | AWS-LC keygen | Ratio | Through materialized matrix | Row sample | Cached row multiply + cache | Cached core | Mulcache |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| AWS Graviton3 | 26.942 us | 19.824 us | 0.74x | 20.848 us | 13.103 us | 2.326 us | 2.093 us | 0.266 us |
| AWS Graviton4 | 23.430 us | 16.356 us | 0.70x | 17.989 us | 11.610 us | 1.904 us | 1.693 us | 0.211 us |

Default dot-product comparison on fixed inputs:

| Platform | Default dot product | Cached row multiply + cache | Cached core |
| --- | ---: | ---: | ---: |
| AWS Graviton3 | 3.178 us | 2.326 us | 2.093 us |
| AWS Graviton4 | 2.613 us | 1.904 us | 1.693 us |

Decision: the remaining K=4 keygen loss is not in `polyvec_mulcache` or the
cached row-multiply kernel. The cached path beats the default dot-product path
on both Graviton lanes, and cache construction is only about 0.21-0.27 us.

The old `materialized-row-multiply` diagnostic remains useful only as a
setup-polluted row. Its G3/G4 medians were 9.305 us and 8.589 us, but the clean
cached row-multiply rows are 2.326 us and 1.904 us. Most of that older row was
deterministic input construction, not multiplication.

Next implementation target: K=4 materialized row sampling. On G3/G4,
`materialized-row-sample` is 13.103 us and 11.610 us, about 63-65% of
forced materialized-through-matrix time. The likely wins are in SHAKE/rejection
parsing and direct row-layout construction, not end-to-end keygen ASM and not
the cached multiply core.

## 2026-06-27 Sampler Phase Diagnostics

Added `diag`-only `mlkem-sampler-phases` rows for K=4 materialized
row-sampling:

- `k=4/triple-xof-setup`
- `k=4/triple-squeeze-3-blocks`
- `k=4/triple-parse-3-blocks`
- `k=4/triple-tail-block`
- `k=4/row-sample-counted`

The bench also prints deterministic K=4 row-sample SHAKE128 rate-block counts
for the sampled matrix entries. Local Apple Silicon count output:

```text
rscrypto-diag auth mlkem_sampler_k4_row_sample_blocks=[3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
```

Local AArch64 quick signal:

| Row | Median |
| --- | ---: |
| `mlkem-keygen-matrix/k=4/materialized-row-sample` | 7.068 us |
| `mlkem-sampler-phases/k=4/triple-xof-setup` | 0.332 us |
| `mlkem-sampler-phases/k=4/triple-squeeze-3-blocks` | 0.999 us |
| `mlkem-sampler-phases/k=4/triple-parse-3-blocks` | 2.145 us |
| `mlkem-sampler-phases/k=4/triple-tail-block` | 1.606 us |
| `mlkem-sampler-phases/k=4/row-sample-counted` | 7.077 us |

Interpretation: counted full-row sampling matches the existing row-sample
time, so the block counters are not materially distorting the measurement.
Every sampled matrix entry consumed three SHAKE128 rate blocks for this fixed
rho. The local split suggests the parser/tail path deserves the next look,
but the G3/G4 numbers should decide because the Graviton row-sample cost is
much higher.

Next G3/G4 run:

```text
RUNNERS: g3,g4
FILTERS: ^mlkem-sampler-phases/k=4/|^mlkem-keygen-matrix/k=4/materialized-row-sample|^(mlkem1024)/keygen/
QUICK: true
```

## 2026-06-27 G3/G4 Sampler Phase Decision

Run
[`#28299930087`](https://github.com/loadingalias/rscrypto/actions/runs/28299930087)
was a targeted G3/G4 quick run at commit
`1b3ec7c7ef40f90f61713a086745cd335c05b011`.

Artifacts were extracted into:

- `benchmark_results/2026-06-27/linux/graviton3/results.txt`
- `benchmark_results/2026-06-27/linux/graviton4/results.txt`

The artifact headers were normalized to the workflow creation time
`2026-06-27 19_49_15` and platform ids `graviton3` / `graviton4`.

Both Graviton lanes printed the same deterministic K=4 sampler block counts:

```text
rscrypto-diag auth mlkem_sampler_k4_row_sample_blocks=[3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
```

Targeted medians:

| Platform | Public keygen | AWS-LC keygen | Ratio | Row sample | Counted row sample | Row/public | Triple setup | Triple squeeze | Triple parse | Triple tail |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| AWS Graviton3 | 27.091 us | 19.828 us | 0.73x | 13.306 us | 13.283 us | 49.1% | 0.667 us | 1.939 us | 1.280 us | 1.294 us |
| AWS Graviton4 | 23.576 us | 16.316 us | 0.69x | 11.716 us | 11.736 us | 49.7% | 0.617 us | 1.779 us | 1.049 us | 1.186 us |

Decision: the sampler diagnosis held on real Graviton lanes. Counted sampling
matches the existing row-sample diagnostic within noise, and every sampled
matrix polynomial takes exactly three SHAKE128 rate blocks for this fixed
`rho`. The keygen gap is therefore not a missing multiply kernel and not a
bad block-count outlier. It is the cost of sampling/parsing the public matrix
and materializing it before the cached row multiply.

Do not write end-to-end owned keygen assembly. Do not chase fewer blocks first;
this input already uses three blocks per polynomial. The next implementation
target is a narrower AArch64 sampler path for the predictable two-full-blocks
plus bounded-third-block shape:

1. Add a diagnostic row that splits K=4 row sampling into first two full
   triple blocks and the third bounded tail block.
2. Prototype a specialized AArch64 third-block parser for `sample_ntt_triple`
   that starts from the known partial fill after two full blocks and writes
   directly into the materialized K=4 row layout.
3. Verify against the scalar sampler tests and fixed FIPS keygen vectors.
4. Rerun G3/G4 with
   `^mlkem-sampler-phases/k=4/|^mlkem-keygen-matrix/k=4/materialized-row-sample|^(mlkem1024)/keygen/`.

## 2026-06-27 Local Sampler Full/Tail Split

Added two more `diag`-only sampler rows:

- `mlkem-sampler-phases/k=4/triple-first-two-blocks`
- `mlkem-sampler-phases/k=4/triple-third-tail-after-two`

The bench now also prints the deterministic fill counts after the first two
triple blocks:

```text
rscrypto-diag auth mlkem_sampler_triple_after_two_fills=[187, 179, 182]
```

Local Apple Silicon quick signal from the final kept diff:

| Row | Median |
| --- | ---: |
| `mlkem-keygen-matrix/k=4/materialized-row-sample` | 7.304 us |
| `mlkem-sampler-phases/k=4/triple-first-two-blocks` | 0.906 us |
| `mlkem-sampler-phases/k=4/triple-third-tail-after-two` | 1.727 us |
| `mlkem-sampler-phases/k=4/triple-tail-block` | 1.598 us |
| `mlkem-sampler-phases/k=4/row-sample-counted` | 7.147 us |

Implementation note: a Rust scalar bounded-tail parser was tried and rejected
locally. It made `materialized-row-sample` and `row-sample-counted` about
7-8% slower on the quick run, so the final code keeps production sampler
behavior unchanged and only adds diagnostics plus a bounded-parser equivalence
test.

Decision: do not land the scalar bounded-tail path. The remaining plausible
optimization is a real AArch64 bounded-tail parser that keeps the current
compact assembly approach but supports capped destination capacity without the
scratch buffer and prefix copy. If that is too large, stop here; the simple
Rust route is worse.

Next G3/G4 run:

```text
RUNNERS: g3,g4
FILTERS: ^mlkem-sampler-phases/k=4/|^mlkem-keygen-matrix/k=4/materialized-row-sample|^(mlkem1024)/keygen/
QUICK: true
```

## 2026-06-27 Local K=4 Quad Row Schedule

The bounded-tail Rust path was the wrong move. The simpler useful change is
to sample each materialized ML-KEM-1024 matrix row with one quad-XOF batch:

- row 0: `(0,0), (1,0), (2,0), (3,0)`
- row 1: `(0,1), (1,1), (2,1), (3,1)`
- row 2: `(0,2), (1,2), (2,2), (3,2)`
- row 3: `(0,3), (1,3), (2,3), (3,3)`

The old triple-plus-single schedule is retained only as a diagnostic
comparator under `mlkem-keygen-matrix/k=4/materialized-row-sample-triple`.

Local Apple Silicon quick signal after promoting the quad schedule to
production:

| Row | Median |
| --- | ---: |
| `mlkem1024/keygen/rscrypto` | 12.579 us |
| `mlkem-keygen-matrix/k=4/materialized-row-sample` | 5.771 us |
| `mlkem-keygen-matrix/k=4/materialized-row-sample-triple` | 7.405 us |

Criterion reported the full keygen row at about `-11.8%` versus the saved
local baseline, but the quick run did not cross p<0.05. The row-sample
comparison is still decisive locally: production quad sampling is about 22%
faster than the retained triple schedule in the same run.

Correctness/validation:

- `cargo check --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth`
- `cargo test --features 'ml-kem,diag' auth::mlkem::portable::tests:: -- --nocapture`
- `cargo bench --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth -- '^mlkem-keygen-matrix/k=4/materialized-row-sample($|-triple$)|^(mlkem1024)/keygen/rscrypto' --quick`
- `cargo clippy --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth -- -D warnings`
- `cargo check --features 'ml-kem'`
- `git diff --check`

Decision: land the quad row schedule before spending another G3/G4 run. The
next paid benchmark should only confirm this production change:

```text
RUNNERS: g3,g4
FILTERS: ^mlkem-keygen-matrix/k=4/materialized-row-sample($|-triple$)|^(mlkem1024)/keygen/
QUICK: true
```

## 2026-06-27 G3/G4 Quad Row Rejection

Run
[`#28302196057`](https://github.com/loadingalias/rscrypto/actions/runs/28302196057)
tested commit `77e0543011f589c164719cd1b28ff00f7d792a78`, which promoted
the quad-XOF row schedule to production and retained the old triple-plus-one
schedule as `mlkem-keygen-matrix/k=4/materialized-row-sample-triple`.

Artifacts were extracted into:

- `benchmark_results/2026-06-27/linux/graviton3/results.txt`
- `benchmark_results/2026-06-27/linux/graviton4/results.txt`

The artifact headers were normalized to workflow creation time
`2026-06-27 21_23_28` and platform ids `graviton3` / `graviton4`.

Targeted medians:

| Platform | Public keygen | AWS-LC keygen | AWS-LC/public | Production row sample | Triple comparator | Production vs triple |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| AWS Graviton3 | 27.694 us | 19.799 us | 0.71x | 13.935 us | 13.263 us | 5.1% slower |
| AWS Graviton4 | 24.364 us | 16.282 us | 0.67x | 12.252 us | 11.728 us | 4.5% slower |

Compared with run `#28299930087`, public keygen regressed:

| Platform | Previous public keygen | New public keygen | Delta |
| --- | ---: | ---: | ---: |
| AWS Graviton3 | 27.091 us | 27.694 us | 2.2% slower |
| AWS Graviton4 | 23.576 us | 24.364 us | 3.3% slower |

Decision: reject the quad-row production promotion for Graviton. The local
Apple Silicon win did not transfer to the paid aarch64 targets that matter
for this optimization. Revert production K=4 materialized row sampling to the
triple-plus-one schedule. If keeping the quad path, keep it as a `diag`-only
negative comparator, not as production code.

## 2026-06-27 Platform Split For K=4 Row Sampling

Implemented a narrow compile-time split instead of throwing away the Apple
Silicon win:

- `aarch64-apple-darwin`: production `materialized-row-sample` uses the
  quad-XOF row schedule.
- every other target, including Linux AArch64 Graviton: production
  `materialized-row-sample` uses the triple-plus-one schedule.
- diagnostic benches now expose both concrete schedules:
  `materialized-row-sample-triple` and `materialized-row-sample-quad`.

Local Apple Silicon quick signal after the split:

| Row | Median |
| --- | ---: |
| `mlkem1024/keygen/rscrypto` | 12.839 us |
| `mlkem-keygen-matrix/k=4/materialized-row-sample` | 5.705 us |
| `mlkem-keygen-matrix/k=4/materialized-row-sample-quad` | 5.692 us |
| `mlkem-keygen-matrix/k=4/materialized-row-sample-triple` | 7.089 us |

That confirms production still follows the faster quad-shaped path on macOS,
while the explicit triple comparator remains slower locally. The Linux
AArch64 build check also passed, so the Graviton/default branch compiles.

Validation:

- `cargo check --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth`
- `cargo test --features 'ml-kem,diag' auth::mlkem::portable::tests::materialized_k4_quad_row_sampler_matches_triple_schedule -- --nocapture`
- `cargo check --features 'ml-kem'`
- `cargo clippy --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth -- -D warnings`
- `cargo bench --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --bench auth -- '^mlkem-keygen-matrix/k=4/materialized-row-sample($|-triple$|-quad$)|^(mlkem1024)/keygen/rscrypto' --quick`
- `cargo check --target aarch64-unknown-linux-gnu --features 'ml-kem'`

Next G3/G4 confirmation run:

```text
RUNNERS: g3,g4
FILTERS: ^mlkem-keygen-matrix/k=4/materialized-row-sample($|-triple$|-quad$)|^(mlkem1024)/keygen/
QUICK: true
```

## 2026-06-27 G3/G4 Platform Split Confirmation

Run
[`#28303684064`](https://github.com/loadingalias/rscrypto/actions/runs/28303684064)
tested commit `a6315a21afa57e7b54da80acfd88b10174d82c78`, which keeps the
quad-XOF row schedule only for `aarch64-apple-darwin` and uses the
triple-plus-one schedule everywhere else.

Artifacts were extracted into:

- `benchmark_results/2026-06-27/linux/graviton3/results.txt`
- `benchmark_results/2026-06-27/linux/graviton4/results.txt`

The artifact headers were normalized to workflow creation time
`2026-06-27 22_26_23` and platform ids `graviton3` / `graviton4`.

Targeted medians:

| Platform | Public keygen | AWS-LC keygen | AWS-LC/public | Production row sample | Triple comparator | Quad comparator |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| AWS Graviton3 | 27.082 us | 19.841 us | 0.73x | 13.249 us | 13.257 us | 13.986 us |
| AWS Graviton4 | 23.571 us | 16.319 us | 0.69x | 11.741 us | 11.739 us | 12.233 us |

Compared with the rejected quad-production run `#28302196057`, the split
restores the lost Graviton performance:

| Platform | Quad-production keygen | Split keygen | Delta |
| --- | ---: | ---: | ---: |
| AWS Graviton3 | 27.694 us | 27.082 us | 2.2% faster |
| AWS Graviton4 | 24.364 us | 23.571 us | 3.3% faster |

Compared with the pre-quad Graviton run `#28299930087`, the split is back at
baseline: G3 `27.091 us` -> `27.082 us`, G4 `23.576 us` -> `23.571 us`.

Decision: keep the platform split. It preserves the Apple Silicon row-sample
win and restores Graviton production behavior to the triple-plus-one schedule.
Do not spend more paid Graviton runs on this scheduling branch.

## 2026-06-27 AWS-LC Keygen Pipeline Diff

Compared local sources for `aws-lc-rs = 1.17.0` / `aws-lc-sys = 0.41.0`
against the current rscrypto ML-KEM-1024 keygen path.

AWS-LC is not avoiding the matrix work and it is not using a magic end-to-end
keygen shape. The keygen pipeline is the same broad structure:

1. expand keygen seed with `G`
2. materialize the full public matrix
3. sample `s` and `e`
4. NTT both vectors
5. compute a mulcache for `s`
6. cached matrix-vector multiply
7. convert `t`, add `e`, reduce
8. pack public and secret-key material

The important AWS-LC details:

- `mlk_indcpa_keypair_derand` calls `mlk_gen_matrix`, samples keygen noise
  with 4-way PRF calls, computes `skpv_cache`, runs `mlk_matvec_mul`, then
  packs.
- `mlk_gen_matrix` materializes matrix entries in row-major groups of four.
  For K=4 this is exactly four x4 matrix-sample batches.
- `mlk_poly_rej_uniform_x4` initially squeezes
  `MLKEM_GEN_MATRIX_NBLOCKS == 3` SHAKE128 blocks per lane, then calls
  `mlk_rej_uniform(... offset = 0, buflen = 3 * rate)`.
- On AArch64, `mlk_rej_uniform(... offset = 0)` dispatches to the native
  `mlk_rej_uniform_asm` path. Tail blocks only happen if three blocks do not
  finish a polynomial; those offset-nonzero blocks fall back to the C path.
- The AArch64 backend also has native NTT, INTT, reduce, tomont, mulcache,
  cached K2/K3/K4 basemul, poly-tobytes, and rejection-sampling kernels.

rscrypto already matches much of that:

- `pke_keygen_expanded` does sample-noise, NTT-noise, matrix-accumulate,
  from-product-domain, and encode.
- Linux AArch64 K=4 production now uses materialized matrix accumulation with
  cached K4 row multiply.
- There is already owned AArch64 block-level rejection ASM for one 168-byte
  SHAKE128 rate block.

The remaining structural difference is narrower: AWS-LC parses the initial
three-block matrix-sample buffer as one capped rejection pass per polynomial.
rscrypto parses one rate block at a time. On the third block, the current
bounded path goes through the scratch/copy cap because the polynomial has less
than one full block of remaining capacity.

Decision: do not write owned ASM for keygen end to end. That would be
ornamental complexity. The next useful prototype is an AWS-LC-shaped initial
three-block sampler:

1. Add a `diag` row for a capped three-block AArch64 parser over one
   polynomial, separate from the current one-block parser loop.
2. Prototype an AArch64 Linux/Darwin `rej_uniform_3blocks` ABI that accepts one
   contiguous `3 * SHAKE128_RATE_BYTES` buffer and writes at most 256
   coefficients directly to the destination.
3. Wire that under a diagnostic `sample_ntt_triple_initial_3blocks_into` path
   first, with scalar/reference equivalence tests and fixed keygen-vector
   coverage.
4. Promote only if local diagnostics move `triple-parse-3-blocks`,
   `triple-third-tail-after-two`, and `materialized-row-sample` in the right
   direction. Do not buy another G3/G4 run until local data says this path is
   real.

2026-07-02 update:

- The three-block capped parser path did not move the row sample. Run
  `28551783178` put `k=4/materialized-row-sample-initial-3blocks` behind the
  production row sampler on both Graviton3 (`13.099us -> 13.229us`) and
  Graviton4 (`11.546us -> 11.581us`).
- The follow-up generic Keccak x3 assembly was worse. Run `28597139827`
  regressed Graviton3 `mlkem768/keygen` from `17.010us` to `21.207us` and
  Graviton4 `mlkem1024/keygen` from `23.395us` to `28.553us`; delete that path
  rather than layering more dispatch around it.
- The next honest target is not matrix scheduling or general-purpose Keccak.
  The Rust-level direct XOF/sampler boundary was tried and rejected locally:
  against clean `4887c16`, Apple Silicon `mlkem1024/keygen` moved from
  `16.561us` to `16.771us`, and `k=4/materialized-row-sample` moved from
  `7.499us` to `7.606us`. The remaining plausible version must be a fused
  ML-KEM matrix-sampler kernel that owns the SHAKE schedule and rejection parse
  together, or a profile showing another phase is now dominant.
- The next production slice is x86_64 first, not another Graviton run. The
  implemented change is an owned AVX2/SSE4.1/SSSE3 public `SampleNTT` parser for
  the materialized row sampler and fused product fallback. Validate on Zen4,
  Zen5, Ice Lake, and Sapphire Rapids with only the ML-KEM keygen/matrix sampler
  quick rows before spending more time on Keccak or AArch64 parser structure.

## 2026-07-02 x86_64 SampleNTT Parser Result

Run
[`#28607261688`](https://github.com/loadingalias/rscrypto/actions/runs/28607261688)
validated the owned x86_64 SIMD `SampleNTT` parser at commit
`5cb1343610d3456501fd5a7c636c27ed872346a1`.

The result confirms the x86 branch of the sampler hypothesis:

| Platform | `mlkem768/keygen` | Delta vs 2026-06-30 rscrypto | Fastest-external status | `mlkem1024/keygen` | Delta vs 2026-06-30 rscrypto | Fastest-external status |
| --- | ---: | ---: | --- | ---: | ---: | --- |
| AMD Zen4 | 10.274 us | 1.23x faster | win vs `libcrux` | 14.368 us | 1.44x faster | win vs `libcrux` |
| AMD Zen5 | 7.333 us | 1.26x faster | win vs `libcrux` | 10.283 us | 1.45x faster | win vs `libcrux` |
| Intel Ice Lake | 12.536 us | 1.28x faster | win vs `libcrux` | 17.930 us | 1.46x faster | win vs `libcrux` |
| Intel Sapphire Rapids | 11.470 us | 1.11x faster | tie vs `aws-lc-rs` | 16.360 us | 1.27x faster | tie vs `aws-lc-rs` |

The targeted run did not include fresh external competitor rows. The status
column compares against the last full Linux external baseline from
`benchmark_results/2026-06-30/linux/*/results.txt`. That is sufficient to keep
the production parser, but the final public scoreboard still needs a full
ML-KEM keygen comparison run.

Ownership result:

- No vendored assembly was removed in this x86_64 slice.
- No vendored assembly was added.
- The new code is owned Rust intrinsics, not `.s` / `.S` assembly, so
  `docs/tasks/asm.md` does not need an entry.
- `scripts/check/asm-ledger.sh` remains `63 assembly files, 15 owned, 48
  external-derived`.

Decision: x86_64 ML-KEM-768/1024 keygen is closed as a targeted Linux loss set:
six wins and two conservative ties, zero losses. The remaining ML-KEM keygen
work is now non-x86: Graviton3/4, RISC-V, and the small Power10
`mlkem1024/keygen` gap.

## 2026-07-02 Non-X86 Matrix Dispatch Cleanup

The next production slice did not add new ASM. It removed stale dispatch into
the generic fused keygen matrix path:

- AArch64 now always uses materialized keygen matrix accumulation. The old
  non-Linux K=4 fused exception was locally slower than materialized.
- RISC-V and PowerPC64 now use materialized keygen matrix accumulation instead
  of the generic fused fallback. Prior artifacts showed their PKE matrix
  materialized path beating fused, and there was no target-owned fused parser
  that justified keeping the slower structure.
- The bench now exposes the real production choice with
  `mlkem-keygen-matrix/k={2,3,4}/default-through-matrix-accumulate`.

Local AArch64 verification after the cleanup:

```text
mlkem-keygen-matrix/k=4/default-through-matrix-accumulate      11.028 us
mlkem-keygen-matrix/k=4/materialized-through-matrix-accumulate 10.9-11.1 us
```

This confirms the default no longer routes to the stale fused path on AArch64.
The remote quick bench must now validate Graviton3, Graviton4, Power10, and
RISC-V with keygen plus the default/fused/materialized matrix rows before any
new target ASM work is justified.

## 2026-07-02 AArch64 Fused Chunk-Loop Follow-Up

Run
[`#28612736993`](https://github.com/loadingalias/rscrypto/actions/runs/28612736993)
confirmed the dispatch cleanup was the right non-x86 default: Power10 closed
and RISC-V improved. Graviton remained the live gap. On Graviton3/4,
`default-through-matrix-accumulate` now tracks materialized, while
`fused-through-matrix-accumulate` is still slower for K=3 and K=4.

The next owned-code slice attacks the fused path's actual low-level waste:

- Linux AArch64 now has
  `rscrypto_mlkem_basemul_accumulate_chunks_aarch64_linux`, an owned ASM
  variable-count loop for contiguous 16-coefficient SampleNTT chunks.
- The previous Linux one-chunk fused helper was removed. The new symbol loads
  Montgomery constants once for all full chunks accepted from a SHAKE block.
- The fused SampleNTT product path now batches full accepted chunks from the
  block scratch into one ASM call.
- Production AArch64 keygen still defaults to materialized until Graviton data
  proves fused is faster; this avoids reintroducing the stale fused dispatch
  mistake.

Validation run locally:

```text
cargo check --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --lib
cargo check --target aarch64-unknown-linux-gnu --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' --lib
cargo check --target aarch64-unknown-linux-gnu --no-default-features --features 'ml-kem' --lib
cargo check --target aarch64-unknown-linux-gnu --no-default-features --features 'ml-kem,diag' --lib
cargo test --features 'hkdf,pbkdf2,ecdsa,ed25519,x25519,ml-kem,diag' auth::mlkem::portable::tests:: --lib
```

The remote quick run should be limited to Graviton3/4, `mlkem768`,
`mlkem1024`, and `mlkem-keygen-matrix`, filtered to public keygen plus
K=3/K=4 default/fused/materialized/windowed matrix rows. If fused still trails
materialized after this owned chunk-loop, stop trying to promote fused and move
to the next lower-level sampler/Keccak target.

2026-07-02 result:

- Run
  [`#28616472237`](https://github.com/loadingalias/rscrypto/actions/runs/28616472237)
  showed the fused chunk-loop was the wrong path. Graviton3/4 public keygen was
  flat, and fused matrix rows regressed by about 1.6-2.2%.
- The Linux AArch64 chunk-loop helper and Rust wiring were deleted. Keep the
  production dispatch on materialized matrix accumulation.

## 2026-07-02 Direct Triple-XOF Sampler Follow-Up

The next low-level AArch64 slice moved from fused multiplication back to the
dominant production sampler:

- Added an AArch64 little-endian `Shake128` triple cursor over three Keccak
  states for ML-KEM public matrix sampling.
- AArch64 `sample_ntt_triple_into` now parses triple rate blocks and remaining
  single-lane tails directly from Keccak state memory.
- The existing owned `rej_uniform` assembly remains the parser backend on
  Linux. No new assembly was added, and no vendored assembly was introduced.
- The old reader-backed triple XOF path is now kept for non-direct targets,
  tests, and `diag` only.

Validation:

```text
cargo check --features ml-kem,sha3
cargo check --features diag,ml-kem,sha3
cargo check --features ml-kem,sha3 --target aarch64-unknown-linux-gnu
cargo clippy --features ml-kem,sha3 -- -D warnings
cargo clippy --features diag,ml-kem,sha3 -- -D warnings
cargo clippy --features ml-kem,sha3 --target aarch64-unknown-linux-gnu -- -D warnings
cargo test --features ml-kem,sha3 --lib shake128_triple_cursor_matches_xof_readers
cargo test --features ml-kem,sha3 --lib sample_ntt_triple_matches_scalar_samplers
```

Required Graviton validation:

```text
RUNNERS: g3,g4
PRIMITIVES/SELECTORS: mlkem768,mlkem1024,mlkem-keygen-matrix
FILTERS: ^mlkem(768|1024)/keygen/(rscrypto|aws-lc-rs)$|^mlkem-keygen-matrix/k=(3|4)/(default-through-matrix-accumulate|materialized-through-matrix-accumulate|fused-through-matrix-accumulate|windowed-through-matrix-accumulate|materialized-row-sample|materialized-row-multiply)$
QUICK: true
```

2026-07-02 result:

Run
[`#28620265132`](https://github.com/loadingalias/rscrypto/actions/runs/28620265132)
showed this direct cursor path was also flat to slightly negative:

| Platform | `mlkem768/keygen` | `mlkem1024/keygen` | K=3 row sample | K=4 row sample |
| --- | ---: | ---: | ---: | ---: |
| Graviton3 | 17.008 us | 26.932 us | 7.0401 us | 13.239 us |
| Graviton4 | 14.990 us | 23.438 us | 6.2859 us | 11.572 us |

Decision: reject and revert the direct triple-XOF cursor production path. It
did not move keygen and made the materialized row-sample diagnostics slightly
slower than the previous same-day materialized baseline. The next AArch64
attempt must stop changing reader structure and either use profile evidence for
a different phase or implement a truly fused ML-KEM matrix sampler that owns
Keccak block generation plus rejection parsing together.

## 2026-07-03 Graviton `perf` Harness Correction

The first direct G3/G4 profiling attempt used the full `auth` Criterion bench.
That was not a valid low-level profiling target for this investigation: it
pulled in external oracle and Criterion dependencies, spent the session on
first-run build/link work, and then the Tailscale SSH path dropped before
useful `perf` reports were recovered. The instances were stopped cleanly.

The corrected profiler target is `src/bin/mlkem_profile.rs`, a real package
binary instead of an example. This matters because Cargo examples are dev
targets and can still pull target-specific `dev-dependencies`; the package bin
uses only normal `rscrypto` dependencies.

Local checks:

```text
cargo check --features ml-kem,diag --bin mlkem_profile
cargo check --features ml-kem,diag --bin mlkem_profile --target aarch64-unknown-linux-gnu
cargo build --profile bench --features ml-kem,diag --bin mlkem_profile
target/release/mlkem_profile --case keygen1024 --seconds 1
target/release/mlkem_profile --case k4-row-sample --seconds 1
```

Remote path:

```text
scripts/bench/mlkem-profile-remote.sh g3
scripts/bench/mlkem-profile-remote.sh g4
```

The helper disables the dev-machine idle-shutdown timer while detached `perf`
runs. Detached profiling has no live SSH session, so leaving the five-minute
idle detector enabled can stop the instance mid-profile. `perf stat` is skipped
by default; `perf stat -d` segfaulted on Graviton3 while reading derived slots
metrics. The helper normalizes ownership on `sudo perf record` outputs before
retrieval. Default profile cases cover end-to-end keygen, production matrix
accumulation, row sampling, row multiply, and the K=4 sampler squeeze/parse
split. Use the resulting `*.perf.report.txt` files to choose the next
implementation target; do not infer the next ASM slice from another public
keygen quick benchmark.

2026-07-03 result:

Artifacts:

```text
benchmark_results/profiles/mlkem/linux-g3-20260703T041249Z/
benchmark_results/profiles/mlkem/linux-g4-20260703T041658Z/
```

The Graviton profile ends the parser-vs-multiply ambiguity. For K=4 keygen,
`sample_matrix_ntt_mul_accumulate_materialized::<4>` is about 56% of cycles on
both G3 and G4, and `keccakf_sha3_x3_hybrid_impl` is about 39-43% of the whole
keygen loop. Inside `k4-matrix-default`, Keccak rises to about 50% on G3 and
53% on G4. Inside `k4-row-sample`, it is about 69% on G3 and 72% on G4.

The owned triple parser is no longer the leading target. It is about 6% of
`keygen1024`, about 7-8% of `k4-matrix-default`, and about 12-13% of
`k4-row-sample`. The cached row multiply kernel is about 6-9% of keygen/matrix.
Both are worth keeping, but neither explains a 0.70x class Graviton keygen
loss.

Decision: the next AArch64 implementation slice must be an owned
ML-KEM-specific matrix XOF/Keccak path, not another forward NTT, fused matrix,
row-multiply, direct cursor, or rejection-parser slice. The target is the
`sample_ntt_triple_into` to `keccakf_sha3_x3_hybrid_impl` path. The security
boundary is clean because public matrix sampling depends on `rho` and matrix
coordinates; its rejection behavior does not branch on secret noise material.

2026-07-03 implementation:

- Added a Linux AArch64 ML-KEM matrix-XOF triple reader.
- Production matrix sampling now initializes and squeezes triple public-matrix
  XOF states with a split schedule: SHA3 CE `x2` for the first two states and
  the tuned scalar permutation for the third.
- The optimized AArch64 `sample_ntt_triple_into` path no longer calls
  `keccakf_sha3_x3_hybrid_impl`; release assembly shows it calling
  `keccakf_sha3_x2_impl` and `keccakf_portable`.
- No parser, row-multiply, fused-matrix, forward-NTT, or inverse-NTT code was
  changed.

This is still a measured-candidate slice, not a declared win. The required
remote signal is G3/G4 movement in `k3-row-sample`, `k4-row-sample`,
`k3-matrix-default`, `k4-matrix-default`, and public `keygen768/keygen1024`.

2026-07-03 quick result:

Run
[`#28666127325`](https://github.com/loadingalias/rscrypto/actions/runs/28666127325)
showed the split `x2 + scalar` matrix-XOF route is a hard regression:

| Platform | `mlkem768/keygen` | `mlkem1024/keygen` | K=3 row sample | K=4 row sample |
| --- | ---: | ---: | ---: | ---: |
| Graviton3 | 19.871 us | 31.726 us | 9.9154 us | 17.999 us |
| Graviton4 | 17.324 us | 27.396 us | 8.7265 us | 15.591 us |

Compared with the previous materialized baseline, public keygen regressed by
about 16-18% and row sampling by about 35-41%. Decision: reject and revert this
route. Removing the hybrid call is not enough; the replacement must preserve
overlap for the third state without the compiler-generated spill pattern.
