kind: llman.sdd.spec
name: "sdd-eval-acp-pipeline"
purpose: "Provide an experimental `llman x sdd-eval` pipeline that can run scripted, repeatable SDD evaluations across multiple workflow styles (`sdd` vs `sdd-legacy`) and multiple ACP agents (Claude Code / Cod"
valid_scope[2]: src/,tests/
requirements[11]{req_id,title,statement}:
  r1,"`llman x sdd-eval` command is available","The CLI MUST provide the experimental subcommand `llman x sdd-eval` with playbook-driven execution."
  r2,Playbooks are managed under the project `.llman/` directory,"By default, `llman x sdd-eval init` MUST create playbooks under `<project>/.llman/sdd-eval/playbooks/`. `llman x sdd-eval run` MUST accept an explicit playbook path, and MUST resolve relative paths from the current working directory."
  r3,"Runs are isolated and stored under `.llman/sdd-eval/runs/<run_id>/`","Each `llman x sdd-eval run` invocation MUST create a new run directory under `<project>/.llman/sdd-eval/runs/<run_id>/`. The run directory MUST contain: - `<run_dir>/manifest.json` (machine-readable run manifest including playbook metadata and variant list) - `<run_dir>/playbook.yaml` (a copy of the playbook used for the run) - `<run_dir>/variants/<variant_id>/workspace/` for each defined variant - `<run_dir>/variants/<variant_id>/logs/` for each defined variant - `<run_dir>/variants/<variant_id"
  r4,Variants combine workflow style and agent preset,"A playbook MUST define one or more `variants` that the workflow can execute. Each variant MUST specify: - an ACP `agent` definition (e.g. `claude-code-acp` or `codex-acp`) - an account preset reference (Claude Code: `llman x cc` group; Codex: `llman x codex` group) The workflow style for this pipeline MUST be new-style SDD only; legacy styles (for example `sdd-legacy`) MUST NOT be supported. Variants MUST be addressable by a stable id (for example `variants.a`, `variants.b`) so that jobs can ref"
  r5,Workflow initialization is performed per variant,"For each variant workspace, the runner MUST initialize/update SDD templates using new-style SDD. In workflow-based playbooks, initialization MUST be exposed as a built-in action (for example `builtin:sdd-eval/sdd.prepare`) so it can be composed as an explicit step."
  r6,ACP agents are launched with preset env injection (without leaking secrets),"The runner MUST support launching ACP agent processes for Claude Code and Codex. For each variant, the runner MUST resolve the referenced preset and inject its environment variables into the spawned ACP agent process. The runner MUST NOT print or write secret values to: - stdout/stderr - playbook files or “resolved playbook” copies - run manifests or per-variant artifacts/logs Only the preset identifier (group name) and environment variable **names** MAY be recorded for debugging."
  r7,The ACP runner is sandboxed to the variant workspace,The ACP client implementation MUST restrict file operations and terminal commands to the variant workspace. It MUST deny attempts to read/write outside the workspace root.
  r8,"Evaluation runs are iteration-bounded and reproducible","The playbook MUST define a fixed maximum iteration count for the SDD loop execution via `sdd_loop.max_iterations`. If `sdd_loop.max_iterations` is omitted, the runner MUST default to 6. If `sdd_loop.max_iterations` is present, it MUST be an integer > 0. The runner MUST stop the SDD loop after the configured number of iterations (no “auto completion” detection in v1)."
  r9,Reports include comparable objective metrics,"`llman x sdd-eval report --run <run_id>` MUST generate a report that includes, per variant: - number of iterations executed - file change summary (counts and sizes) - terminal command summary (commands, exit codes, duration, truncated output)"
  r10,Human scoring export and import are supported,The reporting workflow MUST support exporting a “human scoring pack” (JSON + pointers to relevant artifacts). It MUST support importing a human scoring result file and merging it into the report outputs.
  r11,"AI Judge scoring via OpenAI-compatible API is optional","If enabled in the playbook, the report step MUST be able to call an OpenAI-compatible API using `OPENAI_*` environment variables to produce AI-judge scores. If disabled, the runner MUST NOT require `OPENAI_*` variables."
scenarios[12]{req_id,id,given,when,then}:
  r1,"help-is-available","","the user runs `llman x sdd-eval --help`",the CLI prints help text and exits successfully
  r2,"init-writes-a-yaml-template-playbook","","the user runs `llman x sdd-eval init --name demo` in a project root","`<project>/.llman/sdd-eval/playbooks/demo.yaml` exists and is parseable YAML"
  r3,"run-creates-a-new-run-directory-and-base-layout","","the user runs `llman x sdd-eval run --playbook <path>`","a new `<run_id>` directory exists under `<project>/.llman/sdd-eval/runs/`"
  r4,"missing-variants-fails-loudly","","the playbook has no variants and the user runs `llman x sdd-eval run ...`","the command exits non-zero and explains that at least one variant is required"
  r4,"matrix-references-unknown-variant-fails-loudly","","a job defines `strategy.matrix.variant: [\"a\"]`","the command exits non-zero"
  r5,"variant-workspace-is-initialized-using-new-style-templates","",a variant workspace is prepared for a run,"the variant workspace is initialized using new-style SDD templates (equivalent to `llman sdd init` + `llman sdd update`)"
  r6,"run-artifacts-never-include-an-api-key-value","","the user runs `llman x sdd-eval run ...` with a preset that includes an API key env var","no file under `<project>/.llman/sdd-eval/runs/<run_id>/` contains the raw API key value"
  r7,"path-traversal-is-rejected","",the agent requests reading `../../.ssh/id_rsa`,"the client denies the request and records a non-secret error in the variant log"
  r8,"loop-stops-at-max-iterations","",max iterations is set to 3,"the runner performs at most 3 iterations and then marks the variant as completed-by-limit"
  r9,"report-is-generated-after-a-run","","the user runs `llman x sdd-eval report --run <run_id>`",the command writes a report file under the run directory
  r10,"human-scores-can-be-imported","","the user runs `llman x sdd-eval import-human --run <run_id> --file scores.json`",the report data for `<run_id>` is updated to include the imported scores
  r11,"missing-openai-key-fails-only-when-ai-judge-is-enabled","",`OPENAI_API_KEY` is empty and AI judge is enabled,"`llman x sdd-eval report ...` exits non-zero and explains that `OPENAI_API_KEY` is required for AI judge"
