# P0.3 — Differential testing vs keripy (design)

> Issue: [#27 · P0.3 Differential testing vs keripy](https://github.com/devrandom-labs/cesr/issues/27)
> Milestone: Phase 0 · Instruments
> keripy pin: `v2.0.0.dev5`
> Status: approved design, pre-implementation

## Problem

The `keripy-sync` watcher proves **code-table** parity: cesr defines the same code
points keripy does. It does **not** prove **behavioral** parity — that cesr and
keripy serialize and parse the *same bytes for the same value*. Differential
testing is the real interop guarantee and the strongest correctness guardrail for
the Phase 1 zero-copy / performance refactors that follow.

## Prior art in this repo (do not reinvent)

Two existing patterns solve the same shape and are reused here:

1. **keripy-generated, checked-in, hermetically-replayed vectors.** The committed
   Matter vectors (`src/core/matter/test_vectors*.rs`) carry the header
   *"Generated from KERIpy v2.0.0-dev5"* via a `scripts/generate_test_vectors.py`
   that was **never committed** (it arrived with the initial agency source copy in
   #2). So the repo already has keripy-derived, gate-replayed vectors — but only
   for Matter, with the generator lost and no live-diff loop.
2. **Hermetic-gate + nightly-side-workflow split.** The fuzzing work (#26) replays
   a checked-in corpus deterministically inside `nix flake check` and runs live
   fuzzing in a separate nightly workflow. `keripy-sync` likewise runs keripy in a
   weekly workflow, never in the gate, and takes a keripy checkout via `--keripy`
   (no network in-script), pinning `v2.0.0.dev5`.

P0.3 generalizes pattern (1) to Counter/Indexer/stream and revives the generator,
structured as the two tiers of pattern (2).

## Architecture — two-tier

### Tier 1 · Hermetic gate

Runs inside `nix flake check` via `cargo nextest` (unit tests are already collected
across feature combinations — **no `flake.nix` change required**).

- A checked-in, keripy-generated corpus of vectors for **Matter, Counter v1,
  Counter v2, Indexer, and a composed stream**.
- An **in-crate `#[cfg(all(test, feature = "serder"))]` module** (`src/keripy_diff/`)
  replays each corpus vector two ways:
  - **decode**: parse keripy's `qb64` and `qb2` → assert the reconstructed fields
    (code, raw, soft, index/ondex, count/version) equal the vector's fields.
  - **encode**: build the value from its structured fields → assert the emitted
    `qb64` and `qb2` equal keripy's exact bytes.
- Zero keripy dependency at gate time. This is the byte-exact behavioral oracle.

**Why in-crate `#[cfg(test)]` rather than a `tests/` integration test:** the
primitive counter decode (`stream::parse::parse_counter` / `parse_counter_v2`) is
`pub(crate)` — an external integration test cannot reach it, and exposing it
publicly just for tests would violate the "internal wire-format helpers are
`pub(crate)`" rule. An in-crate test module sees `pub(crate)` and reuses the same
pattern as the existing Matter `test_vectors*.rs` unit tests. Gated on `serder`
because Matter's encode path is `serder::primitives::to_qb64_string` (serder
transitively enables `keri`+`crypto`+`stream`+`core`, exposing every codec the
harness needs). The corpus JSONL is read at test runtime via `std::fs` +
`env!("CARGO_MANIFEST_DIR")`.

### Tier 2 · Nightly live-diff

A separate `.github/workflows/keripy-diff.yml`, non-hermetic — mirrors the existing
nightly deep-fuzz split.

- `scripts/keripy_diff_gen.py` (generalized revival of the lost generator),
  **stdlib + keripy only**, takes `--keripy <checkout>` like `sync.py`; the
  workflow performs the pinned checkout of `v2.0.0.dev5` (no network in-script).
- Input generation: a **boundary-size sweep** (`0, 1, MAX-1, MAX` per code table,
  per CLAUDE.md test rules) plus random fill, plus curated vectors harvested from
  keripy `tests/core/test_*.py` where practical.
- For each input, run keripy encode/decode → emit JSONL vectors → run the diff
  harness against the freshly generated corpus. Any disagreement fails the job.
  On new coverage it opens a PR to extend the committed corpus (as `keripy-sync`
  opens its report PR).

## Corpus format — JSONL data files

`tests/corpus/keripy/{matter,counter_v1,counter_v2,indexer,stream}.jsonl`, one
vector per line:

```json
{"kind":"matter","code":"D","raw":"<hex>","soft":"","qb64":"D...","qb2":"<hex>"}
```

Fields are per-kind (index/ondex for indexer; count/version for counter). Read by
the Rust test via `serde_json`, added to `[dev-dependencies]` so the diff test has
it regardless of crate feature selection.

**Why JSONL over the existing generated-`.rs` convention:** an append-growable
corpus diffs cleanly line-by-line, is language-neutral (Python writes, Rust reads),
and avoids ever-growing multi-thousand-line generated Rust files. The existing
Matter `TestVector` `.rs` files remain as the unit-level tier; this is the new
cross-implementation tier.

## Pinning

keripy **`v2.0.0.dev5`** — the single tag already used by `keripy-sync` and the
existing Matter vectors. One source of truth; bump deliberately.

## Error handling & acceptance

- Any decode-mismatch, encode-mismatch, or parse-failure on a keripy-valid vector
  is a hard test failure that prints the offending vector.
- A genuine cesr↔keripy disagreement is filed as a bug issue rather than silently
  tolerated (satisfies the AC "zero disagreements on the covered surface, or files
  bugs for any").
- Input-validation failures (malformed corpus line) are distinct from
  parity-mismatch failures — different assertion sites, different messages.

## Modules / boundaries

| Unit | Purpose | Depends on |
|------|---------|------------|
| `src/keripy_diff/mod.rs` | `#[cfg(all(test, feature="serder"))]` harness; JSONL loader + shared assert helpers | corpus files, `serde_json`, codec API |
| `src/keripy_diff/{matter,counter,indexer,stream}.rs` | Per-kind decode+encode replay | corpus, codec API |
| `tests/corpus/keripy/*.jsonl` | Frozen keripy-generated oracle vectors | — |
| `scripts/keripy_diff_gen.py` | Tier-2 generator (`--keripy` path; boundary + random) | a keripy checkout |
| `.github/workflows/keripy-diff.yml` | Nightly: pinned checkout → regen → test → PR | generator, keripy tag |

## Codec entry points (resolved — the two former open items)

Confirmed against the source; the harness calls exactly these:

| Kind | Decode (qb64) | Encode → qb64 | qb2 |
|------|---------------|---------------|-----|
| Matter | `MatterBuilder::new().from_qualified_base64(bytes)` (pub) / `.from_qualified_base2(bytes)` | build via `MatterBuilder::new().with_code(MatterCode::from_str(c)?).with_raw(raw)?.with_soft(soft)?.build()?` then `serder::primitives::to_qb64_string(&m)?` (serder-gated) | `stream::qb64_to_qb2(qb64)` (no native Matter qb2 encode) |
| Counter v1 | `stream::parse::parse_counter(bytes)` → `(CounterCodeV1, u32, rest)` (**`pub(crate)`**) | `stream::encode::encode_counter_v1(code, count)?` (pub) | `stream::qb64_to_qb2(qb64)` |
| Counter v2 | `stream::parse::parse_counter_v2(bytes)` (**`pub(crate)`**) | `stream::encode::encode_counter_v2(code, count)?` (pub) | `stream::qb64_to_qb2(qb64)` |
| Indexer | `IndexerBuilder::new().from_qb64(bytes)` → `(Indexer, consumed)` (pub) | build via `IndexerBuilder::new().with_code(c).with_index(i)?.with_raw(raw)?` then `.to_qb64()` (pub) | native `.to_qb2()` (pub) |

Consequences baked into the design above: (1) the `pub(crate)` counter decode forces
the **in-crate `#[cfg(test)]`** harness; (2) Matter/Counter have **no native qb2
encode**, so their qb2 direction is checked via the public `qb64_to_qb2` transcoder,
while Indexer uses its native `to_qb2()`.

## Open item (resolved during implementation)

**Composed-stream shape.** Define the exact multi-primitive stream both keripy and
cesr round-trip (a Counter frame + N Matters), and how the harness walks it — decode
via the public `stream::parse_group` / `groups` API, cross-checked with
`qb64_to_qb2`/`qb2_to_qb64`. Exact framing fixed in the plan's stream task.

## Non-goals

- No `pyo3` / in-process linking and no subprocess-per-input in the gate (both
  break `nix flake check` hermeticity — explicitly rejected).
- No change to the existing unit-level Matter `test_vectors*.rs`.
- No coverage beyond Matter / Counter v1+v2 / Indexer / one composed stream in this
  iteration.

## Acceptance criteria (from #27)

- [ ] Research recommendation captured (esp. the hermeticity decision) — this doc +
      an issue comment.
- [ ] Harness finds zero disagreements on the covered surface (or files bugs).
- [ ] Integrated into `nix flake check` (Tier 1 via nextest) + nightly workflow
      (Tier 2).
