# Adversarial testing system — design summary (gate 3)

*Updated: 2026-05-29. Source: `adv_system.md`, `adversarial.md`, KPop gates 1–3 (`exp_log_20260529_180346_jysxg20w_g1.md`, `exp_log_20260529_201409_5qtwdrku_g1.md`, `exp_log_20260529_231503_sucjfj0y_g1.md`).*

## Goal

A **generic adversarial testing system** that prevents code, tests, linters, and static analyses from **overfitting** to benchmark fixtures. It runs non-interactively alongside malvin agent sessions and asserts **structural law** on witness-graph outcomes—not scalar fit on a fixed benchmark tree.

Overfitting targets (same failure, different surface):

- Special-cased tests
- Brittle tests
- Analysis rules tuned to benchmark dirname/crate/argv inventories
- Code that passes an eval or linter without capturing its intent

## Core model

Static analysis maps **definitions** \(D\) and **witnesses** to per-unit outcomes \(c: D \to O\) under a deterministic bound. An **adversarial obligation** is a falsifiable predicate on \(c\) that must hold under input transforms \(T\).

Two oracle-free mechanisms:

| Mechanism | Shape | Role |
|-----------|-------|------|
| **Metamorphic testing** | Source fixture \(x\), follow-up \(x'=T(x)\), relation \(R(f(x), f(x'))\) | Rename, graft, surface signatures, negation anti-MRs |
| **Property-based testing** | Single frozen input; algebraic property on output | Monotonicity, idempotence, order invariance, operator-pathology sequences |

**Hard constraint:** Production analysis \(f\) is deterministic. Randomness only in test generation.

## What we falsified

### Gate 1

| Hypothesis | Result | Implication |
|------------|--------|-------------|
| Definition-path rename MR catches all dirname inventory | Falsified | Add **witness-path bijection** MR and **`invert`** token substitution |
| Rename + graft tiers make `compose` optional | Falsified | **`malvin adversarial compose`** is mandatory for MR composition bugs |
| Single-op algebra PBT catches order-dependent walks | Falsified | Add **operator-pathology PBT** on frozen graphs |
| Scalar benchmark metrics suffice in unit gates | Rejected by research | Keep scalars in **ops ablation**, not unit adversarial |

### Gate 2

| Hypothesis | Result | Implication |
|------------|--------|-------------|
| Fixture-only `invert` hunts all inventory dependence | Falsified | **`invert` operates on fixture AND source** transform domains |
| Point predicates on \(c(d)\) suffice for `channel_collapse` | Falsified | **`oracle: differential` only** — vector relation on channel metadata |
| Transform-only shrunk bundles replay without registry | Falsified | Bundles = transform cert + **topology digest registry** (optional embedded tree fallback) |
| Witness-path-only bijection subsumes def-path rename | Falsified | Keep **both** rename MR classes (confirms gate 1) |

### Gate 3

| Hypothesis | Result | Implication |
|------------|--------|-------------|
| In-memory graph specs suffice for subprocess plugins | Falsified | **Filesystem materialization** before plugin call; walkdir discovers graft shells absent from spec |
| Void-graph rows use point predicates on \(c(d)\) | Falsified | **Empty outcome-vector MR** across graft shells at \(\|D\|=0\) |
| `check-registry` schema alone rejects vacuous rows | Falsified | **Null-plugin non-vacuity**: rows null plugin satisfies are invalid |
| Single `topology_digest` covers source-domain invert | Falsified | Shrunk bundles carry **`topology_digest` + `config_digest`** |

## Smell-class registry

Each row is an executable contract (`smell_registry.toml`):

- `id`, `principle`, `invariant`, `counterexample_shape`, `depends_on`, `target_modules`
- `oracle: differential` for rows like `channel_collapse` — **no point predicate on \(c(d)\)**; require \(\Delta(f_{\text{full}}, f_{\text{incremental}})=0\) on partition-channel metadata vector
- `counterexample_shape: void` rows declare **empty outcome-vector MR**, not point predicates

Initial catalog: `dirname_inventory`, `filename_inventory`, `cli_argv_inventory`, `workspace_crate_inventory`, `repo_root_vocabulary`, `numeric_staircase`, `channel_collapse`, `non_local_context`. Consumer repos extend; malvin does not hard-code domain smells.

**Synthetic path rule:** Counterexample fixtures use neutral trees (`acme/`, `vendor/`, `widget_tests/`). Benchmark tokens appear only via `invert` / audit manifest, not committed fixtures.

## Tier stack (`malvin adversarial exec`)

Cheap guards first for green-build wall-clock; `compose` may invert ordering when hunting violations.

```
0  Authorization + token manifest
1  Registry meta-completeness (+ null-plugin non-vacuity, invent #9)
2  Literal audit (grep forbidden tokens in analysis source)
3  Point predicates + negation anti-MRs
4  Algebra PBT + operator-pathology on frozen graphs
5  Graft MR (non-local shell isolation; void-graph empty-vector variant)
6  Surface topology MR (transition signatures)
7  Rename MR (global segment bijection on definition AND witness paths)
8  Optional topology fuzz
9  Path-gate / channel guards
—  Ablation on benchmark suite (ops only; scalar metrics)
```

## `malvin adversarial` subcommands

| Subcommand | Purpose |
|------------|------|
| `check-registry` | Schema, acyclic `depends_on`, meta-completeness, **null-plugin non-vacuity** |
| `exec` | Run tier stack; emit violations + shrunk bundles |
| `compose` | Search composed transform chains to falsify MRs |
| `invert` | Forbidden follow-ups on **fixtures and analysis source**; prove inventory dependence |
| `diff-rows` | Incremental re-run from git diff on `target_modules` |

**Inputs:** `--registry`, `--principles`, `--token-manifest`, `--bound`, `--plugin`, `--budget materializations=N`, `--target-smell`.

**Outputs:** Non-zero exit on violation; report with `smell_id`, transform chain, minimal topology; shrunk bundles under `.malvin/adversarial/shrunk/<smell_id>/`.

### Materialization harness (gate 3 spike)

| Component | Role |
|-----------|------|
| **Filesystem materialization** | Write full temp-repo tree before subprocess plugin invocation; graft shells must exist on disk |
| **`topology_registry/`** | Digest → canonical neutral fixture (fixture bytes only; excludes plugin config) |
| **`config_registry/`** (or inline in bundle) | Config digest → plugin config snapshot for source-domain invert replay |
| **Void-graph canonical entry** | Zero-definition topology with graft shells only; tests empty outcome-vector invariance |

Subprocess plugins that walk the filesystem cannot run on in-memory graph descriptors alone. Library plugins may accept structured input, but the malvin subprocess contract assumes materialized paths.

### `invert` transform domains

| Domain | Transform \(T\) | Catches |
|--------|-----------------|---------|
| Fixture | Substitute manifest tokens into neutral repo tree paths | Path-string inventory in \(x\) |
| Source | Substitute manifest tokens into plugin config / blocklists / allowlists | Inventory smuggled into \(f\) |
| Dual polarity (optional) | Stability-under-all-substitutions mode | Negation anti-MR as primary certifier |

Fixture-only invert is insufficient (gate 2 falsification).

### Shrunk-bundle schema

```json
{
  "smell_id": "non_local_context",
  "transform_chain": ["graft:vendor@d=2", "rename:phi_abc"],
  "topology_digest": "sha256:…",
  "config_digest": "sha256:…",
  "violation_witness": { "field": "outcome_vector", "shell": "benchmark_token" },
  "embedded_topology": null
}
```

Replay: `materialize(topology_digest)` from topology registry + `materialize(config_digest)` from config registry, apply `transform_chain`, re-run plugin. Optional `embedded_topology` for offline replay without registry lookup.

### Analysis plugin contract

```
Input:  materialized temp repo + bound/config (filesystem paths)
Output: structured map (definition id → outcome) + metadata (partition, channel)
```

Deterministic: identical outputs on identical frozen fixtures. **Subprocess plugins require filesystem materialization** (gate 3).

## Agent integration

| Workflow | Hook |
|----------|------|
| `malvin constrain` | Gate may require clean `adversarial exec` or `compose --budget N` before `KPOP_SOLVED` |
| `malvin code` | Pre-gate when plan touches `target_modules` |
| `malvin tidy` | Surface adversarial failures alongside kiss |
| `malvin invent` | Ideas name smell rows they might violate; `invert` falsifies before code lands |

**Inversion-primary mode:** Default hunt via `compose`/`invert` (both transform domains); `exec` certifies shrunk replays.

Shrunk bundles are **not** auto-committed; human or follow-up `malvin constrain` crystallizes permanent tests.

## Artifact layout

```
.malvin/
  adversarial/
    shrunk/<smell_id>/          # violation bundles
    topology_registry/          # fixture digest → canonical neutral tree
    config_registry/            # config digest → plugin config snapshot
smell_registry.toml             # optional malvin default
token_manifest.toml             # audit-derived forbidden tokens
```

Consumer repos: principles doc, audit findings, fixtures, plugin implementations.

## Implementation bounds

Adversary implementation should respect repo complexity rules (`.kissconfig`). Kiss research (`../kiss/coverage/research`) motivates calibration against runtime gold but does not replace structural MRs in unit gates.

## Next gates (adv_system.md cycle)

Remaining invent/falsify cycles should:

1. Implement temp-repo materialization harness (filesystem write + dual digest registries)
2. Prototype `check-registry` with null-plugin non-vacuity on a sample TOML
3. Add void-graph canonical fixture to `topology_registry/` and graft tier spike
4. Prototype dual-domain `invert` on a kiss calibration plugin

## References

- `adversarial.md` — full formal spec
- Chen et al. metamorphic testing; Claessen & Hughes QuickCheck; Su et al. differential testing
- Kiss research summaries: Alves & Visser, Aniche et al., Böhme ICSE 2023, CoverageEval, CodeFlow
- Gate 3 invent log: `.malvin/logs/20260529_231533_eq0uzp7n`
