# P0.3 — Differential testing vs keripy — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Prove byte-exact behavioral parity between cesr and keripy for Matter, Counter (v1+v2), Indexer, and a composed stream, enforced hermetically in `nix flake check` plus a nightly live-diff workflow.

**Architecture:** Two tiers. Tier 1 is an in-crate `#[cfg(all(test, feature = "serder", feature = "std"))]` harness (`src/keripy_diff/`) that replays a checked-in JSONL corpus of keripy-generated vectors — decode keripy's bytes → assert cesr fields, and encode cesr fields → assert keripy's bytes. Zero keripy dependency at gate time; runs under the gate's existing `--all-features` nextest. Tier 2 is `scripts/keripy_diff_gen.py` (runs against a pinned keripy `v2.0.0.dev5` checkout) plus `.github/workflows/keripy-diff.yml`, which regenerates the corpus, diffs, and opens a PR on new coverage.

**Tech Stack:** Rust 2024 (stable 1.95.0), `serde` + `serde_json` (already pulled by the `serder` feature — no new dependency), Python 3 stdlib + keripy (`keri` package) for the generator, GitHub Actions for the nightly job.

**Spec:** `docs/superpowers/specs/2026-07-01-p0.3-differential-testing-design.md`
**Issue:** https://github.com/devrandom-labs/cesr/issues/27
**Branch:** `test/p0.3-differential-testing` (already created; spec committed)

---

## File Structure

| Path | Responsibility | Created/Modified |
|------|----------------|------------------|
| `src/lib.rs` | Declare the test-only harness module | Modify (add one gated `mod` line) |
| `src/keripy_diff/mod.rs` | `DiffVector` serde struct, JSONL loader, hex helper, submodule decls | Create |
| `src/keripy_diff/matter.rs` | Matter decode+encode replay | Create |
| `src/keripy_diff/counter.rs` | Counter v1+v2 decode+encode replay | Create |
| `src/keripy_diff/indexer.rs` | Indexer decode+encode replay | Create |
| `src/keripy_diff/stream.rs` | Composed-stream replay | Create |
| `tests/corpus/keripy/matter.jsonl` | Frozen keripy Matter vectors | Create (generated) |
| `tests/corpus/keripy/counter_v1.jsonl` | Frozen keripy Counter v1 vectors | Create (generated) |
| `tests/corpus/keripy/counter_v2.jsonl` | Frozen keripy Counter v2 vectors | Create (generated) |
| `tests/corpus/keripy/indexer.jsonl` | Frozen keripy Indexer vectors | Create (generated) |
| `tests/corpus/keripy/stream.jsonl` | Frozen keripy composed-stream vectors | Create (generated) |
| `scripts/keripy_diff_gen.py` | Tier-2 generator (`--keripy` checkout → JSONL) | Create |
| `.github/workflows/keripy-diff.yml` | Nightly regen + diff + PR | Create |

**Corpus JSONL line schema** (one object per line):
```json
{"kind":"matter","code":"D","raw":"<hex>","soft":"","qb64":"D...","qb2":"<hex>"}
```
Per-kind extra fields: `index`,`ondex` (indexer); `count` (counter); `elements` (stream, an array of primitive objects). `raw` and `qb2` are lowercase hex; `qb64` is the literal string.

---

## Task 1: Harness scaffold — module, loader, `DiffVector`

**Files:**
- Modify: `src/lib.rs` (after line 30, the `utils` module block)
- Create: `src/keripy_diff/mod.rs`

- [ ] **Step 1: Write the failing test** — inline fixture, no real corpus yet.

Create `src/keripy_diff/mod.rs`:
```rust
//! Differential-testing harness vs keripy (P0.3, issue #27).
//!
//! Replays a checked-in, keripy-generated JSONL corpus and asserts cesr
//! agrees with keripy byte-for-byte on both encode and decode. Test-only and
//! `serder`-gated so every codec path (incl. Matter encode via
//! `serder::primitives::to_qb64_string`) is in scope.

use serde::Deserialize;

mod counter;
mod indexer;
mod matter;
mod stream;

#[derive(Debug, Deserialize)]
pub(crate) struct DiffVector {
    pub kind: String,
    #[serde(default)]
    pub code: String,
    #[serde(default)]
    pub raw: String,
    #[serde(default)]
    pub soft: String,
    pub index: Option<u32>,
    pub ondex: Option<u32>,
    pub count: Option<u32>,
    #[serde(default)]
    pub qb64: String,
    #[serde(default)]
    pub qb2: String,
    #[serde(default)]
    pub elements: Vec<DiffVector>,
}

pub(crate) fn from_hex(s: &str) -> Vec<u8> {
    assert!(s.len() % 2 == 0, "odd-length hex: {s}");
    (0..s.len())
        .step_by(2)
        .map(|i| u8::from_str_radix(&s[i..i + 2], 16).expect("valid hex byte"))
        .collect()
}

pub(crate) fn load(kind: &str) -> Vec<DiffVector> {
    let path = format!(
        "{}/tests/corpus/keripy/{kind}.jsonl",
        env!("CARGO_MANIFEST_DIR")
    );
    let text = std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("read {path}: {e}"));
    text.lines()
        .filter(|l| !l.trim().is_empty())
        .map(|l| serde_json::from_str::<DiffVector>(l).unwrap_or_else(|e| panic!("parse `{l}`: {e}")))
        .collect()
}

#[cfg(test)]
mod scaffold_tests {
    use super::{from_hex, DiffVector};

    #[test]
    fn parses_one_line_and_decodes_hex() {
        let line = r#"{"kind":"matter","code":"D","raw":"deadbeef","qb64":"Dxx","qb2":"0102"}"#;
        let v: DiffVector = serde_json::from_str(line).unwrap();
        assert_eq!(v.kind, "matter");
        assert_eq!(v.code, "D");
        assert_eq!(from_hex(&v.raw), vec![0xde, 0xad, 0xbe, 0xef]);
        assert_eq!(from_hex(&v.qb2), vec![0x01, 0x02]);
        assert_eq!(v.soft, "");
        assert!(v.index.is_none());
    }
}
```

Wire it into `src/lib.rs` immediately after the `utils` module block (line 30):
```rust
#[cfg(all(test, feature = "serder", feature = "std"))]
mod keripy_diff;
```

The four submodules (`counter`,`indexer`,`matter`,`stream`) are declared now but their files are created empty in this task so the crate compiles:
```bash
: > src/keripy_diff/counter.rs
: > src/keripy_diff/indexer.rs
: > src/keripy_diff/matter.rs
: > src/keripy_diff/stream.rs
```

- [ ] **Step 2: Run test to verify it fails (compiles, asserts pass — but confirm it runs under serder)**

Run: `nix develop --command cargo test --all-features keripy_diff::scaffold_tests -- --nocapture`
Expected: PASS (this task has no real corpus dependency yet). If it does not compile, fix before proceeding.

- [ ] **Step 3: Confirm the module is invisible without serder (no_std/default builds stay green)**

Run: `nix develop --command cargo build`
Expected: PASS — `keripy_diff` is `#[cfg(all(test, feature = "serder", ...))]`, so a default build never sees it.

- [ ] **Step 4: Commit**

```bash
git add src/lib.rs src/keripy_diff/
git commit -m "test(diff): scaffold keripy differential harness (loader + DiffVector)"
```

---

## Task 2: Tier-2 generator — `scripts/keripy_diff_gen.py`

**Files:**
- Create: `scripts/keripy_diff_gen.py`

- [ ] **Step 1: Confirm keripy's API against the pinned checkout (no assumptions)**

Before writing, read these files at keripy `v2.0.0.dev5` to confirm constructor kwargs and the `.qb64`/`.qb2` attributes:
- `src/keri/core/coring.py` — `Matter(raw=, code=)`, attrs `.qb64`, `.qb2`
- `src/keri/core/counting.py` — `Counter(...)` for CESR 1.0 (`CtrDex_1_0`) and 2.0 (`CtrDex_2_0`), attrs `.qb64`, `.qb2`, and the count kwarg name
- `src/keri/core/indexing.py` — `Indexer(raw=, code=, index=, ondex=)`, attrs `.qb64`, `.qb2`

- [ ] **Step 2: Write the generator**

```python
#!/usr/bin/env python3
"""Generate keripy differential-test corpus (JSONL) from a keripy checkout.

Deterministic given (--keripy, --seed): a boundary-size sweep plus a fixed-seed
random fill over each code table. No network — the caller provides a keripy
checkout (mirrors tools/keripy-sync/sync.py). Pin: keripy v2.0.0.dev5.
"""
import argparse
import json
import os
import random
import sys
from pathlib import Path


def emit(fh, obj):
    fh.write(json.dumps(obj, separators=(",", ":"), sort_keys=True) + "\n")


def gen_matter(rng, Matter, MtrDex, out):
    # One vector per fixed-size code, at a deterministic random raw of its size.
    with (out / "matter.jsonl").open("w") as fh:
        for name, code in sorted(vars(MtrDex).items()):
            if name.startswith("_") or not isinstance(code, str):
                continue
            try:
                sizes = Matter.Sizes[code]
            except (KeyError, AttributeError):
                continue
            rawsize = getattr(sizes, "rawsize", None)
            if rawsize is None or callable(rawsize):
                continue  # skip variable-size codes in the seed sweep
            raw = bytes(rng.randrange(256) for _ in range(rawsize))
            try:
                m = Matter(raw=raw, code=code)
            except Exception:
                continue
            emit(fh, {
                "kind": "matter", "code": code, "raw": raw.hex(),
                "soft": "", "qb64": m.qb64, "qb2": m.qb2.hex(),
            })


def gen_counter(rng, Counter, dex, version, fname, out):
    with (out / fname).open("w") as fh:
        for name, code in sorted(vars(dex).items()):
            if name.startswith("_") or not isinstance(code, str):
                continue
            for count in _counts(rng):
                try:
                    c = Counter(code=code, count=count, version=version)
                except TypeError:
                    try:
                        c = Counter(code=code, count=count)
                    except Exception:
                        continue
                except Exception:
                    continue
                emit(fh, {
                    "kind": "counter", "code": code, "count": count,
                    "qb64": c.qb64, "qb2": c.qb2.hex(),
                })


def _counts(rng):
    # Boundary sweep per CLAUDE.md test rules + one random mid value.
    return [0, 1, 4094, 4095, rng.randrange(1, 4096)]


def gen_indexer(rng, Indexer, IdrDex, out):
    with (out / "indexer.jsonl").open("w") as fh:
        for name, code in sorted(vars(IdrDex).items()):
            if name.startswith("_") or not isinstance(code, str):
                continue
            try:
                sizes = Indexer.Sizes[code]
                rawsize = sizes.rawsize if hasattr(sizes, "rawsize") else None
            except (KeyError, AttributeError):
                continue
            if rawsize is None:
                continue
            raw = bytes(rng.randrange(256) for _ in range(rawsize))
            for index in (0, 1, 63):
                try:
                    ix = Indexer(raw=raw, code=code, index=index)
                except Exception:
                    continue
                emit(fh, {
                    "kind": "indexer", "code": code, "raw": raw.hex(),
                    "index": index, "ondex": getattr(ix, "ondex", None),
                    "qb64": ix.qb64, "qb2": ix.qb2.hex(),
                })


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--keripy", required=True, type=Path, help="path to a keripy checkout")
    ap.add_argument("--out", required=True, type=Path, help="corpus output dir")
    ap.add_argument("--seed", type=int, default=0xC E5R & 0xFFFF)  # replaced below
    args = ap.parse_args()

    sys.path.insert(0, str((args.keripy / "src").resolve()))
    from keri.core.coring import Matter, MtrDex
    from keri.core.counting import Counter
    from keri.core import counting, indexing
    from keri.core.indexing import Indexer, IdrDex

    args.out.mkdir(parents=True, exist_ok=True)
    rng = random.Random(args.seed)

    gen_matter(rng, Matter, MtrDex, args.out)
    gen_counter(rng, Counter, counting.CtrDex_1_0, "CESR", "counter_v1.jsonl", args.out)
    gen_counter(rng, Counter, counting.CtrDex_2_0, "CESR", "counter_v2.jsonl", args.out)
    gen_indexer(rng, Indexer, IdrDex, args.out)
    print(f"wrote corpus to {args.out}", file=sys.stderr)


if __name__ == "__main__":
    main()
```

> Note: the `--seed` default above is a placeholder token to force a review — replace the `default=` expression with a literal `default=42` when writing the file (the `0xC E5R` text is intentionally invalid so this line cannot be committed unread). The `version` argument to `Counter` (`"CESR"` / the `Vrsn_2_0` object) is one of the items to confirm in Step 1; adjust to whatever `counting.py` requires.

- [ ] **Step 3: Confirm it imports keripy and produces JSONL (needs a keripy checkout)**

```bash
git clone --depth 1 --branch v2.0.0.dev5 https://github.com/WebOfTrust/keripy /tmp/keripy
python3 -m venv /tmp/keripy-venv && /tmp/keripy-venv/bin/pip install -e /tmp/keripy
/tmp/keripy-venv/bin/python scripts/keripy_diff_gen.py --keripy /tmp/keripy --out /tmp/corpus
head -1 /tmp/corpus/matter.jsonl
```
Expected: each file has one JSON object per line with non-empty `qb64` and `qb2`.

- [ ] **Step 4: Commit the generator**

```bash
git add scripts/keripy_diff_gen.py
git commit -m "feat(diff): keripy corpus generator (scripts/keripy_diff_gen.py)"
```

---

## Task 3: Generate and commit the frozen corpus

**Files:**
- Create: `tests/corpus/keripy/{matter,counter_v1,counter_v2,indexer}.jsonl`

- [ ] **Step 1: Regenerate straight into the repo** (reuses the `/tmp/keripy` checkout from Task 2)

```bash
/tmp/keripy-venv/bin/python scripts/keripy_diff_gen.py \
  --keripy /tmp/keripy --out tests/corpus/keripy
wc -l tests/corpus/keripy/*.jsonl
```
Expected: four non-empty files.

- [ ] **Step 2: Sanity-check the corpus is valid JSONL**

Run: `python3 -c "import json,sys; [json.loads(l) for f in sys.argv[1:] for l in open(f)]" tests/corpus/keripy/*.jsonl && echo OK`
Expected: `OK`

- [ ] **Step 3: Commit the corpus**

```bash
git add tests/corpus/keripy/
git commit -m "test(diff): checked-in keripy corpus (matter/counter/indexer) @v2.0.0.dev5"
```

---

## Task 4: Matter differential replay

**Files:**
- Modify: `src/keripy_diff/matter.rs`

- [ ] **Step 1: Write the failing test**

`src/keripy_diff/matter.rs`:
```rust
use core::str::FromStr;

use crate::core::matter::MatterBuilder;
use crate::core::matter::code::MatterCode;
use crate::keripy_diff::{from_hex, load};
use crate::serder::primitives::to_qb64_string;
use crate::stream::qb64_to_qb2;

#[test]
fn matter_differential_vs_keripy() {
    let vectors = load("matter");
    assert!(!vectors.is_empty(), "matter corpus is empty");

    for v in &vectors {
        let code = MatterCode::from_str(&v.code)
            .unwrap_or_else(|_| panic!("unknown matter code {}", v.code));
        let raw = from_hex(&v.raw);

        // decode: keripy qb64 -> cesr fields
        let m = MatterBuilder::new()
            .from_qualified_base64(v.qb64.as_bytes())
            .unwrap_or_else(|e| panic!("matter {} qb64 decode: {e:?}", v.code));
        assert_eq!(m.raw(), raw.as_slice(), "matter {} raw", v.code);
        assert_eq!(m.soft(), v.soft, "matter {} soft", v.code);
        assert_eq!(m.code(), &code, "matter {} code", v.code);

        // encode: cesr fields -> keripy qb64
        let built = MatterBuilder::new()
            .with_code(code)
            .with_raw(&raw)
            .unwrap_or_else(|e| panic!("matter {} with_raw: {e:?}", v.code))
            .build()
            .unwrap_or_else(|e| panic!("matter {} build: {e:?}", v.code));
        assert_eq!(
            to_qb64_string(&built).unwrap(),
            v.qb64,
            "matter {} encode qb64",
            v.code
        );

        // qb2: transcode keripy qb64 -> qb2 and cross-check the frozen bytes
        assert_eq!(
            qb64_to_qb2(v.qb64.as_bytes()).unwrap(),
            from_hex(&v.qb2),
            "matter {} qb2",
            v.code
        );
        // decode keripy qb2 -> cesr fields
        let m2 = MatterBuilder::new()
            .from_qualified_base2(from_hex(&v.qb2))
            .unwrap_or_else(|e| panic!("matter {} qb2 decode: {e:?}", v.code));
        assert_eq!(m2.raw(), raw.as_slice(), "matter {} raw from qb2", v.code);
    }
}
```

> Implementation note: if `build()` is not reachable directly after `with_raw` for a given code's typestate, or a soft-bearing code appears in the corpus, add a `with_soft(&v.soft)` call guarded by `if !v.soft.is_empty()`. Resolve against `src/core/matter/builder.rs` typestate transitions — the compiler will pinpoint it. Confirm the `MatterCode` import path (`crate::core::matter::code::MatterCode` vs a re-export) via `grep -rn "pub use.*MatterCode" src/core/matter/`.

- [ ] **Step 2: Run to verify it fails first without the impl wired (it is wired — so run for PASS)**

Run: `nix develop --command cargo test --all-features matter_differential_vs_keripy -- --nocapture`
Expected: PASS. If a code fails to decode/encode, that is a **real cesr↔keripy disagreement** — stop and file a bug issue per the spec's acceptance criteria before continuing.

- [ ] **Step 3: Commit**

```bash
git add src/keripy_diff/matter.rs
git commit -m "test(diff): Matter differential replay vs keripy"
```

---

## Task 5: Counter v1 + v2 differential replay

**Files:**
- Modify: `src/keripy_diff/counter.rs`

- [ ] **Step 1: Write the failing test**

`src/keripy_diff/counter.rs`:
```rust
use crate::core::counter::{CounterCodeV1, CounterCodeV2};
use crate::keripy_diff::{from_hex, load};
use crate::stream::encode::{encode_counter_v1, encode_counter_v2};
use crate::stream::parse::{parse_counter, parse_counter_v2};
use crate::stream::qb64_to_qb2;

#[test]
fn counter_v1_differential_vs_keripy() {
    let vectors = load("counter_v1");
    assert!(!vectors.is_empty(), "counter_v1 corpus is empty");
    for v in &vectors {
        let code = CounterCodeV1::from_hard(&v.code)
            .unwrap_or_else(|_| panic!("unknown counter_v1 code {}", v.code));
        let count = v.count.unwrap_or_else(|| panic!("counter_v1 {} missing count", v.code));

        // encode: (code, count) -> keripy qb64
        let qb64 = encode_counter_v1(code, count)
            .unwrap_or_else(|e| panic!("counter_v1 {} encode: {e:?}", v.code));
        assert_eq!(qb64, v.qb64.as_bytes(), "counter_v1 {} qb64", v.code);

        // decode: keripy qb64 -> (code, count)
        let (dcode, dcount, rest) = parse_counter(v.qb64.as_bytes())
            .unwrap_or_else(|e| panic!("counter_v1 {} decode: {e:?}", v.code));
        assert_eq!(dcode, code, "counter_v1 {} decoded code", v.code);
        assert_eq!(dcount, count, "counter_v1 {} decoded count", v.code);
        assert!(rest.is_empty(), "counter_v1 {} trailing bytes", v.code);

        // qb2 cross-check
        assert_eq!(
            qb64_to_qb2(v.qb64.as_bytes()).unwrap(),
            from_hex(&v.qb2),
            "counter_v1 {} qb2",
            v.code
        );
    }
}

#[test]
fn counter_v2_differential_vs_keripy() {
    let vectors = load("counter_v2");
    assert!(!vectors.is_empty(), "counter_v2 corpus is empty");
    for v in &vectors {
        let code = CounterCodeV2::from_hard(&v.code)
            .unwrap_or_else(|_| panic!("unknown counter_v2 code {}", v.code));
        let count = v.count.unwrap_or_else(|| panic!("counter_v2 {} missing count", v.code));

        let qb64 = encode_counter_v2(code, count)
            .unwrap_or_else(|e| panic!("counter_v2 {} encode: {e:?}", v.code));
        assert_eq!(qb64, v.qb64.as_bytes(), "counter_v2 {} qb64", v.code);

        let (dcode, dcount, rest) = parse_counter_v2(v.qb64.as_bytes())
            .unwrap_or_else(|e| panic!("counter_v2 {} decode: {e:?}", v.code));
        assert_eq!(dcode, code, "counter_v2 {} decoded code", v.code);
        assert_eq!(dcount, count, "counter_v2 {} decoded count", v.code);
        assert!(rest.is_empty(), "counter_v2 {} trailing bytes", v.code);

        assert_eq!(
            qb64_to_qb2(v.qb64.as_bytes()).unwrap(),
            from_hex(&v.qb2),
            "counter_v2 {} qb2",
            v.code
        );
    }
}
```

> Confirm exact import paths: `grep -rn "pub fn parse_counter\|pub fn encode_counter\|pub use.*CounterCode" src/stream src/core/counter`. `parse_counter*` are `pub(crate)` (reachable in-crate). `CounterCodeV1/V2` re-export at `crate::core::counter`.

- [ ] **Step 2: Run to verify**

Run: `nix develop --command cargo test --all-features 'counter_v' -- --nocapture`
Expected: PASS. Any mismatch is a real disagreement → file a bug before continuing.

- [ ] **Step 3: Commit**

```bash
git add src/keripy_diff/counter.rs
git commit -m "test(diff): Counter v1+v2 differential replay vs keripy"
```

---

## Task 6: Indexer differential replay

**Files:**
- Modify: `src/keripy_diff/indexer.rs`

- [ ] **Step 1: Write the failing test**

`src/keripy_diff/indexer.rs`:
```rust
use core::str::FromStr;

use crate::core::indexer::{IndexedSigCode, IndexerBuilder};
use crate::keripy_diff::{from_hex, load};

#[test]
fn indexer_differential_vs_keripy() {
    let vectors = load("indexer");
    assert!(!vectors.is_empty(), "indexer corpus is empty");
    for v in &vectors {
        let code = IndexedSigCode::from_str(&v.code)
            .unwrap_or_else(|_| panic!("unknown indexer code {}", v.code));
        let raw = from_hex(&v.raw);
        let index = v.index.unwrap_or_else(|| panic!("indexer {} missing index", v.code));

        // decode: keripy qb64 -> cesr fields
        let (ix, consumed) = IndexerBuilder::new()
            .from_qb64(v.qb64.as_bytes())
            .unwrap_or_else(|e| panic!("indexer {} decode: {e:?}", v.code));
        assert_eq!(ix.raw(), raw.as_slice(), "indexer {} raw", v.code);
        assert_eq!(consumed, v.qb64.len(), "indexer {} consumed", v.code);

        // encode: cesr fields -> keripy qb64 + qb2
        let built = IndexerBuilder::new()
            .with_code(code)
            .with_index(index)
            .unwrap_or_else(|e| panic!("indexer {} with_index: {e:?}", v.code))
            .with_raw(&raw)
            .unwrap_or_else(|e| panic!("indexer {} with_raw: {e:?}", v.code));
        assert_eq!(built.to_qb64(), v.qb64, "indexer {} encode qb64", v.code);
        assert_eq!(built.to_qb2(), from_hex(&v.qb2), "indexer {} encode qb2", v.code);
    }
}
```

> Confirm: `grep -rn "pub use\|pub enum IndexedSigCode\|impl FromStr for IndexedSigCode\|pub fn from_qb64\|pub fn with_indices" src/core/indexer`. If `IndexedSigCode` has no `FromStr`, map `v.code` via the code enum's own constructor (e.g. `from_hard`/`from_str` equivalent — use whatever the enum exposes). For `Both`-mode codes carrying a distinct `ondex`, extend the corpus vector + use `with_indices(index, ondex)`; the seed generator emits `ondex` already.

- [ ] **Step 2: Run to verify**

Run: `nix develop --command cargo test --all-features indexer_differential_vs_keripy -- --nocapture`
Expected: PASS. Any mismatch is a real disagreement → file a bug before continuing.

- [ ] **Step 3: Commit**

```bash
git add src/keripy_diff/indexer.rs
git commit -m "test(diff): Indexer differential replay vs keripy"
```

---

## Task 7: Composed-stream differential replay

**Files:**
- Modify: `src/keripy_diff/stream.rs`
- Modify: `scripts/keripy_diff_gen.py` (add `gen_stream`)
- Modify: `tests/corpus/keripy/stream.jsonl` (regenerate)

- [ ] **Step 1: Define the stream shape and extend the generator**

The composed stream is a V2 attachment group counter framing N controller-indexed signatures (Counter `-A`/`CtrDex_2_0` + Indexer elements) — the smallest stream exercising counter-frames-primitives. Add to `scripts/keripy_diff_gen.py`:
```python
def gen_stream(rng, Counter, Indexer, IdrDex, out):
    # One vector: a controller-indexed-sig group of 2 Ed25519 sigs.
    code = getattr(IdrDex, "Ed25519_Sig", None) or getattr(IdrDex, "Ed25519")
    rawsize = Indexer.Sizes[code].rawsize
    sigs = [Indexer(raw=bytes(rng.randrange(256) for _ in range(rawsize)),
                    code=code, index=i) for i in range(2)]
    ctr = Counter(code="-A", count=len(sigs))  # confirm ctor in counting.py
    qb64 = ctr.qb64 + "".join(s.qb64 for s in sigs)
    qb2 = ctr.qb2 + b"".join(s.qb2 for s in sigs)
    with (out / "stream.jsonl").open("w") as fh:
        emit(fh, {
            "kind": "stream", "qb64": qb64, "qb2": qb2.hex(),
            "elements": [{"kind": "indexer", "code": code, "index": i,
                          "raw": s.raw.hex() if hasattr(s, "raw") else "",
                          "qb64": s.qb64, "qb2": s.qb2.hex()}
                         for i, s in enumerate(sigs)],
        })
```
Call `gen_stream(rng, Counter, Indexer, IdrDex, args.out)` in `main()`. Regenerate:
```bash
/tmp/keripy-venv/bin/python scripts/keripy_diff_gen.py --keripy /tmp/keripy --out tests/corpus/keripy
```

- [ ] **Step 2: Write the failing test**

`src/keripy_diff/stream.rs`:
```rust
use crate::keripy_diff::{from_hex, load};
use crate::stream::{qb2_to_qb64, qb64_to_qb2};
use crate::stream::parse::parse_counter_v2;

#[test]
fn stream_differential_vs_keripy() {
    let vectors = load("stream");
    assert!(!vectors.is_empty(), "stream corpus is empty");
    for v in &vectors {
        // qb64 <-> qb2 transcode agrees with keripy's frozen bytes, both ways
        assert_eq!(
            qb64_to_qb2(v.qb64.as_bytes()).unwrap(),
            from_hex(&v.qb2),
            "stream qb64->qb2"
        );
        assert_eq!(
            qb2_to_qb64(&from_hex(&v.qb2)).unwrap(),
            v.qb64.as_bytes(),
            "stream qb2->qb64"
        );

        // the outer counter parses and its count == number of elements
        let (_code, count, rest) = parse_counter_v2(v.qb64.as_bytes())
            .unwrap_or_else(|e| panic!("stream counter decode: {e:?}"));
        assert_eq!(count as usize, v.elements.len(), "stream element count");
        assert!(!rest.is_empty(), "stream body after counter");
    }
}
```

> If the public group walker (`crate::stream::{parse_group_v2, groups_v2, CesrGroup}`) is a better fit for asserting each element, extend this test to decode the group and cross-check each `elements[i].qb64`. Confirm signatures: `grep -rn "pub fn parse_group_v2\|pub fn groups_v2" src/stream`.

- [ ] **Step 3: Run to verify**

Run: `nix develop --command cargo test --all-features stream_differential_vs_keripy -- --nocapture`
Expected: PASS.

- [ ] **Step 4: Commit**

```bash
git add src/keripy_diff/stream.rs scripts/keripy_diff_gen.py tests/corpus/keripy/stream.jsonl
git commit -m "test(diff): composed-stream differential replay vs keripy"
```

---

## Task 8: Nightly live-diff workflow

**Files:**
- Create: `.github/workflows/keripy-diff.yml`

- [ ] **Step 1: Write the workflow** (model on `.github/workflows/keripy-sync.yml` and `fuzz.yml`)

```yaml
# keripy-diff — nightly behavioral parity check vs keripy.
# Regenerates the differential corpus from a pinned keripy checkout, runs the
# in-crate diff harness against it, and opens a PR if the corpus changed.
name: keripy-diff

on:
  schedule:
    - cron: "0 4 * * *"
  workflow_dispatch:

concurrency:
  group: keripy-diff
  cancel-in-progress: true

permissions:
  contents: write
  pull-requests: write

jobs:
  diff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/checkout@v4
        with:
          repository: WebOfTrust/keripy
          ref: v2.0.0.dev5
          path: keripy
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install keripy
        run: python -m pip install -e ./keripy
      - name: Regenerate corpus
        run: python scripts/keripy_diff_gen.py --keripy ./keripy --out tests/corpus/keripy
      - uses: DeterminateSystems/nix-installer-action@main
      - name: Run differential harness
        run: nix develop --command cargo test --all-features keripy_diff -- --nocapture
      - name: Open PR on corpus drift
        uses: peter-evans/create-pull-request@v6
        with:
          branch: keripy-diff/corpus
          commit-message: "test(diff): refresh keripy differential corpus"
          title: "test(diff): keripy differential corpus refresh"
          body: |
            Nightly `keripy-diff` regenerated the corpus from keripy `v2.0.0.dev5`.
            Review the vector delta; a change means keripy's output moved or new
            coverage was found. If the harness step failed above, a real
            cesr↔keripy disagreement exists — do not merge; file a bug.
```

- [ ] **Step 2: Lint the workflow**

Run: `nix develop --command actionlint .github/workflows/keripy-diff.yml`
Expected: no errors.

- [ ] **Step 3: Commit**

```bash
git add .github/workflows/keripy-diff.yml
git commit -m "ci(diff): nightly keripy differential parity workflow"
```

---

## Task 9: Full gate + close-out

- [ ] **Step 1: Run the full gate**

Run: `nix flake check`
Expected: all checks pass, including `cesr-nextest` (which runs `--all-features`, so the `keripy_diff` harness executes). If a new file is untracked, `git add` it first (the gate needs staged files).

- [ ] **Step 2: Push and open the PR**

```bash
git push -u origin test/p0.3-differential-testing
gh pr create --title "test: P0.3 differential testing vs keripy (#27)" \
  --body "Closes #27. Two-tier differential harness: hermetic checked-in keripy corpus replayed in the gate (\`src/keripy_diff/\`, serder-gated) + nightly regen/diff workflow. Covers Matter, Counter v1+v2, Indexer, composed stream. See docs/superpowers/specs/2026-07-01-p0.3-differential-testing-design.md."
```

- [ ] **Step 3: Attach the PR/issue to the CESR project board** (org Project #5, per team convention).

- [ ] **Step 4: Post a close-out comment on #27** summarizing coverage and any bugs filed.

---

## Self-Review notes

- **Spec coverage:** Matter (Task 4), Counter v1+v2 (Task 5), Indexer (Task 6), composed stream (Task 7), hermetic gate integration (Task 1 + Task 9, `--all-features` nextest), nightly Tier-2 (Task 8), generator + pinned keripy (Tasks 2–3). All spec sections mapped.
- **Deliberate review gates:** two lines are intentionally un-committable until read — the `--seed` default token in Task 2 and the `version`/`Counter` ctor kwargs — because they depend on keripy API details that MUST be confirmed against the pinned checkout (Task 2 Step 1), not assumed.
- **Type consistency:** loader `load(kind)` and `from_hex` (Task 1) are reused verbatim by Tasks 4–7. `DiffVector` fields (`code`,`raw`,`soft`,`index`,`ondex`,`count`,`qb64`,`qb2`,`elements`) cover every kind. Codec entry points match the spec's resolved table.
- **Known compile-time unknowns (resolved by TDD + compiler):** Matter builder typestate (`build()` after `with_raw`, optional `with_soft`), `IndexedSigCode` constructor name, exact re-export paths. Each task names the `grep` to confirm and the compiler pinpoints the rest.
