# Test requirements: `atproto-devtool test labeler`

This document maps every acceptance criterion from the design plan to the concrete test(s) that verify it, names the phase in which the test is authored, and lists the fixtures and verification approach.

**Design plan:** `docs/design-plans/2026-04-13-test-labeler.md`
**Implementation plan:** `docs/implementation-plans/2026-04-13-test-labeler/`

---

## Conventions

- **Unit tests** live in the same file as the code they test (`#[cfg(test)] mod tests { ... }`).
- **Integration tests** live in `tests/*.rs` and link against `src/lib.rs`. Naming follows the stage: `labeler_identity.rs`, `labeler_http.rs`, `labeler_subscription.rs`, `labeler_crypto.rs`, `labeler_end_to_end.rs`.
- **Snapshot tests** use `insta` with `cargo insta review`/`cargo insta test --accept`. Snapshots live under `tests/snapshots/`.
- **CLI-level tests** drive the compiled binary via `assert_cmd::Command::cargo_bin("atproto-devtool")`.
- **Subscription tests** use `#[tokio::test(flavor = "current_thread", start_paused = true)]` so `tokio::time::advance` drives both the fake's inter-frame sleeps and the stage's budget timer deterministically.
- **Identity-stage report rendering** builds a local `miette::GraphicalReportHandler` per `LabelerReport::render` call rather than relying on the process-global handler, avoiding parallel-test races.
- **Fixture hygiene:** on-disk fixtures under `tests/fixtures/labeler/` are captured once from real responses (recorded in commit messages), then hand-mutated for failure cases. Each fixture file names the AC it exercises.
- **Do not mock external crates.** Fakes implement the narrow trait(s) the stage depends on (`RawHttpTee`, `WebSocketClient`, `PlcLogFetcher`, `IdentityResolver`) so tests can drive them without touching the network.

---

## AC coverage matrix

Each AC is owned by exactly one verification test unless stated otherwise. Phase column names the phase that writes the test; some diagnostics from earlier phases are verified end-to-end again in Phase 6's `insta` suite.

### test-labeler.AC1 — CLI skeleton and invocation modes

| AC | Test | Phase | Kind | Notes |
|----|------|-------|------|-------|
| AC1.1 | `handle_target_runs_all_stages` | 6 | `insta` end-to-end | Fake backends for all four stages; handle resolver returns a known DID; assert snapshot shows identity + HTTP + subscription + crypto stages with `[OK]` glyphs. |
| AC1.2 | `did_target_runs_all_stages` | 6 | `insta` end-to-end | Same fakes, invoked with bare `did:plc:…` — handle resolver must not be called. |
| AC1.3 | `url_target_skips_identity_and_crypto` | 6 | `insta` end-to-end | Invoked with `https://labeler.example/`; assert identity and crypto stages are `[SKIP]` with the documented reasons, HTTP + subscription run. |
| AC1.4 | `url_with_did_cross_checks_endpoint` | 6 | `insta` end-to-end | Two variants: match (all-pass) and mismatch (`SpecViolation` on endpoint comparison). |
| AC1.5 | `invalid_target_exits_two` | 1 | CLI (`assert_cmd`) | `test labeler not a valid target` → exit code 2, clap error text. |
| AC1.6 | `help_lists_all_flags` | 1 | CLI (`assert_cmd`) | `test labeler --help` stdout contains `--did`, `--subscribe-timeout`, `--verbose`, `--no-color`, `<TARGET>`. |

### test-labeler.AC2 — Identity-layer checks

| AC | Test | Phase | Kind | Notes |
|----|------|-------|------|-------|
| AC2.1 | `valid_did_doc_and_labeler_record_passes` | 3 | Integration | Fakes return fixture DID doc with `#atproto_labeler` service + parseable k256 key, plus fixture labeler record with non-empty `policies.labelValues`. Assert `Outcome::Pass` on both identity checks. |
| AC2.2 | `labeler_record_with_label_values_passes` | 3 | Integration | Covered as the record-half of AC2.1's fixture; asserted independently. |
| AC2.3 | `missing_service_entry_spec_violation` | 3 | `insta` | Fixture DID doc with no `#atproto_labeler`. Snapshot asserts `[FAIL]`, diagnostic `NamedSource` = the DID JSON, `#[label]` span highlights the `service` array. |
| AC2.4 | `missing_signing_key_spec_violation` | 3 | `insta` | Fixture DID doc with service but no labeler verification method. |
| AC2.5 | `invalid_service_endpoint_spec_violation` | 3 | `insta` | Fixture DID doc with `serviceEndpoint: "ftp://bad"`. |
| AC2.6 | `missing_labeler_record_spec_violation` | 3 | Integration | Fake PDS returns 404; assert distinct `SpecViolation` code from transport failure. |
| AC2.7 | `empty_label_values_spec_violation` | 3 | `insta` | Fixture labeler record with `policies.labelValues: []`; span highlights `policies`. |
| AC2.8 | `dns_failure_is_network_error` | 3 | Integration | Fake identity resolver returns a DNS-style error; assert `NetworkError` and that run exit code is `0` on its own. |

### test-labeler.AC3 — HTTP-layer checks

| AC | Test | Phase | Kind | Notes |
|----|------|-------|------|-------|
| AC3.1 | `well_formed_query_labels_decodes` | 4 | Integration | `FakeRawHttpTee` returns valid JSON; assert typed decode via `atrium_api::com::atproto::label::query_labels::Output` and `Outcome::Pass`. |
| AC3.2 | `cursor_pagination_round_trip_passes` | 4 | Integration | Fake returns distinct page for cursor; assert stage called `query_labels(None)` then `query_labels(Some(&cursor))` and that responses differ. |
| AC3.3 | `empty_labels_produces_advisory` | 4 | Integration | Fake returns `{ "labels": [] }`; assert pass with attached `Advisory` ("labeler has no published labels"). |
| AC3.4 | `malformed_query_labels_spec_violation` | 4 | `insta` | Fake returns JSON missing `labels` field; diagnostic `NamedSource` is the response bytes; snapshot assertion. |
| AC3.5 | `ignored_cursor_spec_violation` | 4 | Integration | Fake returns the same page regardless of cursor; assert `SpecViolation` on pagination check. |
| AC3.6 | `unreachable_endpoint_network_error` | 4 | Integration | Fake returns transport error; assert `NetworkError` and that schema check is `Skipped`, not `Fail`. |

### test-labeler.AC4 — Subscription-layer checks

| AC | Test | Phase | Kind | Notes |
|----|------|-------|------|-------|
| AC4.1 | `backfill_completes_within_budget_passes` | 5 | `insta` | Fake script: 3 frames + 600ms idle; `tokio::time::advance` drives the clock. |
| AC4.2 | `backfill_exceeds_budget_triggers_live_tail` | 5 | `insta` | Two-script fake: budget-exceeding first connection, clean live-tail second. |
| AC4.3 | `empty_stream_advisories` | 5 | `insta` | No frames at all; assert backfill `[WARN]` and live-tail `[SKIP]` with matching reason strings, exit code `0`. |
| AC4.4 | `malformed_frame_emits_spec_violation` | 5 | `insta` | 1 valid + 1 malformed frame; assert `[FAIL] subscription::frame_decode` with source bytes = the malformed fixture. |
| AC4.5 | `error_frame_malformed_payload_spec_violation` | 5 | `insta` | First frame is `op: -1` with malformed `#info` payload. |
| AC4.6 | `unreachable_endpoint_network_error` (sub) | 5 | Integration | Fake returns transport error on connect; assert `NetworkError`, summary calls out one network error, exit `0`. |
| AC4.7 | `subscribe_timeout_below_floor_rejected` | 5 (helper) + 1 (CLI) | Unit + CLI | Unit test on `parse_subscribe_timeout` asserts `"500ms"` → `Err` containing `"at least 1 second"`; `"0"` → any `Err`. Optional `assert_cmd` end-to-end with `500ms` asserts exit 2 and `"at least 1 second"` substring. |

### test-labeler.AC5 — Crypto-layer checks

| AC | Test | Phase | Kind | Notes |
|----|------|-------|------|-------|
| AC5.1 | `current_key_verifies_all_labels_passes` | 6 | Integration | Fake `PlcLogFetcher` not called; assert stage passes without fetching history. |
| AC5.2 | `rotated_out_key_verifies_via_plc_history` | 6 | Integration | Hand-crafted fixture: label signed by a historic key; fake PLC log returns the history; assert `Pass` + `Advisory` listing key ids. |
| AC5.3 | `empty_labeler_skips_crypto_stage` | 6 | Integration | HttpFacts has empty `first_page`; assert `Skipped("labeler published no labels; nothing to verify")` and that it does not affect exit code. |
| AC5.4 | `did_web_current_key_mismatch_fails` | 6 | `insta` | Fixture did:web labeler with bad signature; assert `Fail` rollup diagnostic listing the current key id and the "no rotation history" message. |
| AC5.5 | `did_plc_current_and_historic_mismatch_fails` | 6 | `insta` | did:plc labeler, none of {current, historic} verify; assert diagnostic lists every key id tried. |
| AC5.6 | `canonicalization_failure_distinct_from_sig_fail` | 6 | Integration | Fixture label with invalid CBOR in the record field; assert per-label `Fail` with a diagnostic code distinct from the signature-mismatch code. |
| AC5.7 | `plc_log_fetch_failure_is_network_error` | 6 | Integration | Fake PLC log returns transport error; assert `NetworkError` and that labels failing against the current key are reported as `Fail` (not cascade-hidden). |

**Plus** the canonicalizer regression fixture — `tests/fixtures/labeler/crypto/reference_label.cbor` — is a byte-exact golden. `canonicalize_label_for_signing` tests assert the output equals the golden file; drift in canonicalization breaks this test. Written in Phase 6, subcomponent A.

### test-labeler.AC6 — Cross-cutting reporting and exit semantics

| AC | Test | Phase | Kind | Notes |
|----|------|-------|------|-------|
| AC6.1 | `all_pass_run_exits_zero_with_ok_glyphs` | 6 | `insta` end-to-end | Full fake stack; snapshot the rendered report and assert exit code `0`. |
| AC6.2 | `spec_violation_run_exits_one` | 6 | `insta` end-to-end | Run with a fixture producing one `SpecViolation` in each stage; assert footer severity breakdown and exit `1`. |
| AC6.3 | `network_error_only_run_exits_zero` | 6 | `insta` end-to-end | All stages produce only `NetworkError`; assert separate network-error count and exit `0`. |
| AC6.4 | `advisory_only_run_exits_zero` | 6 | `insta` end-to-end | Only advisories; exit `0`. |
| AC6.5 | `skipped_checks_render_reason_strings` | 6 | `insta` end-to-end | Verified alongside AC1.3's URL-mode snapshot. |
| AC6.6 | `no_color_suppresses_ansi_keeps_glyphs` | 6 | CLI (`assert_cmd`) | Run with `NO_COLOR=1`; assert stdout contains `[OK]`/`[FAIL]` glyphs but no ESC sequences. |
| AC6.7 | `verbose_flag_emits_debug_tracing_to_stderr` | 6 | CLI (`assert_cmd`) | Run with `--verbose`; assert stderr contains at least one `DEBUG atproto_devtool::` line and rendered report on stdout is unchanged from the non-verbose snapshot. |
| AC6.8 | `bootstrap_failure_exits_two` | 6 | CLI (`assert_cmd`) | Run with an unparseable CLI arg; assert exit code `2`, distinct from `1`. Pairs with AC1.5. |

---

## Fixtures

All fixtures live under `tests/fixtures/labeler/<stage>/`. Each fixture file begins with a header comment naming the AC(s) it supports.

- **identity/**: `valid_did_doc.json`, `missing_service.json`, `missing_signing_key.json`, `invalid_service_endpoint.json`, `empty_label_values.json`, `valid_labeler_record.json`.
- **http/**: `first_page.json`, `second_page.json`, `empty_page.json`, `missing_labels_field.json`, `ignores_cursor_page.json`.
- **subscription/**: `frame_info_ok.bin`, `frame_labels_ok.bin`, `frame_malformed_cbor.bin`, `frame_error_malformed_info.bin`. Each `.bin` is the raw WebSocket payload.
- **crypto/**: `reference_label.cbor` (canonicalizer golden), `label_signed_current.json`, `label_signed_historic.json`, `label_invalid_cbor.json`, `plc_log_with_rotation.json`, `did_web_doc_no_history.json`.
- **end-to-end/**: composed from the above; no unique fixture files.

Capture procedure (documented in Phase 2 for identity, Phase 4 for HTTP, Phase 5 for subscription, Phase 6 for crypto):

1. Run the stage against a known-good production labeler, save raw bytes.
2. Hand-mutate into failure variants with a single minimal edit per variant.
3. Commit with a message naming the source and the mutation.

---

## Verification gates

Before the plan is considered complete, ALL of the following must pass from a clean worktree:

1. `cargo fmt --check` — exit 0, no output.
2. `cargo clippy -- -D warnings` — clean.
3. `cargo build` — succeeds.
4. `cargo test` — all tests pass (unit + integration).
5. `cargo insta test` — no pending snapshot changes.
6. Every AC in the table above resolves to at least one passing test under (4) or (5).

The end-to-end Phase 6 `insta` suite is the final gate: if any cross-cutting AC (AC6.x) regresses but phase-local tests still pass, the report-rendering or exit-code wiring has drifted and must be fixed before merge.

---

## Manual verification

A small set of ACs are difficult to exercise without real network IO. These are verified manually by running the compiled binary against known production labelers and recording the output in the PR description. None of these are required for CI green, but they are required before merging to `main`:

- **AC1.1–AC1.4:** Run `cargo run -- test labeler moderation.bsky.app` and variants (handle, DID, URL, URL + `--did`). Confirm each stage produces output consistent with the fake-backed snapshots.
- **AC2.8:** Run against an intentionally-bad handle (`cargo run -- test labeler does-not-resolve.invalid`) and confirm `NetworkError` classification and exit code `0`.
- **AC3.6:** Run against an offline endpoint (`cargo run -- test labeler https://127.0.0.1:1/`) and confirm HTTP-stage `NetworkError`.
- **AC4.6:** As AC3.6 but confirming the subscription-stage classification.
- **AC5.2:** Run against `moderation.bsky.app` if/when it has rotated signing keys historically. If PLC log contains no rotation history for the labeler, this path is exercised only via the unit test with a crafted fixture.
- **AC6.7:** Run `cargo run -- test labeler --verbose moderation.bsky.app 2>/tmp/stderr.txt` and confirm `/tmp/stderr.txt` contains `DEBUG` lines mentioning `atproto_devtool::identity`, `::http`, `::subscription`, and `::crypto`.

Record these manual checks in the PR description with the exact commands run and the output observed.
