kind: llman.sdd.spec
name: "sdd-eval-workflow-dsl"
purpose: "Define the workflow/jobs/steps DSL for `llman x sdd-eval` playbooks, including job dependencies, matrix variant expansion, built-in actions, minimal interpolation, and `run` step sandboxing."
valid_scope[2]: src/,tests/
requirements[9]{req_id,title,statement}:
  r1,Playbook Uses a Workflow/Jobs/Steps Model,"The `llman x sdd-eval` playbook MUST be a YAML document that defines a workflow to execute. The playbook MUST be a YAML mapping. The playbook MUST define: - `task` (the shared evaluation task) - `variants` (the AB groups to evaluate) - `workflow.jobs` (the jobs and steps to run) `workflow` MUST be a YAML mapping and MUST define: - `jobs` (a mapping of job ids to job definitions) Each `job_id` under `workflow.jobs` MUST match `^[a-zA-Z][a-zA-Z0-9_-]*$`. Each job definition MUST be a YAML mapping "
  r2,Variants Are Addressable by Stable Id and Expandable by Matrix,"The playbook MUST define `variants` as a mapping from a stable `variant_id` to a variant configuration. Each `variant_id` MUST: - match `^[a-zA-Z][a-zA-Z0-9_-]*$` - be used as the directory name under `<run_dir>/variants/<variant_id>/` Each variant configuration MUST define: - `agent`: an ACP agent definition (kind/preset + optional command/args) The workflow style MUST be new-style SDD only; legacy styles (for example `sdd-legacy`) MUST NOT be supported, and playbooks MUST NOT include a `varian"
  r3,Job `needs` Dependencies Are Resolved Deterministically,"The runner MUST resolve job `needs` dependencies deterministically. Jobs MAY declare `needs: [job_id...]` to depend on other jobs. The runner MUST: - reject a `needs` entry that references an unknown job id - reject dependency cycles - execute jobs in a deterministic topological order - when multiple jobs are runnable, use YAML declaration order as a stable tie-breaker"
  r4,Step Kind Is Either `uses` or `run`,"Each job MUST define an ordered list of `steps`. Each step MUST specify exactly one of: - `uses`: reference a built-in action - `run`: execute a local command Each step MUST be a YAML mapping. Allowed step keys are: - `name` (optional string) - `uses` (string) and `with` (optional mapping) - `run` (string) and `cwd` (optional string) If `uses` is present: - `with` MAY be present - `cwd` MUST NOT be present If `run` is present: - `cwd` MAY be present - `with` MUST NOT be present Unknown step keys"
  r5,"Built-In Actions Are Supported","The runner MUST support the following built-in actions (stable identifiers): - `builtin:sdd-eval/workspace.prepare` - `builtin:sdd-eval/sdd.prepare` - `builtin:sdd-eval/acp.sdd-loop` - `builtin:sdd-eval/report.generate` If a step references an unknown built-in action, the runner MUST fail loudly."
  r6,"Built-In Actions Have Stable Semantics and Outputs","Each built-in action MUST operate only within its sandbox root: - for matrix-expanded jobs: the variant workspace root (`<run_dir>/variants/<variant_id>/workspace/`) - for non-matrix jobs: the run directory root (`<run_dir>/`) Built-in actions MUST NOT read/write outside their sandbox root. The following stable semantics MUST be implemented: - `builtin:sdd-eval/workspace.prepare` - MUST copy the project root into the variant workspace. - MUST NOT copy: - `.git/`, `.llman/`, `target/`, `node_modu"
  r7,Minimal Context Interpolation Is Supported,"The runner MUST support simple string interpolation using the syntax `${{ <path> }}`. Interpolation MUST be a pure string substitution (no expressions, operators, or functions). Whitespace around `<path>` inside `${{ ... }}` MAY be present and MUST be ignored. The runner MUST apply interpolation before executing a step, at minimum for: - `run` step fields (`run`, and `cwd` if present) - the `uses` value - all string values inside `with` (recursively) At minimum, the following paths MUST be suppo"
  r8,`run` Steps Execute Allowlisted Commands Under Sandbox,"The runner MUST support `run` steps for workflow jobs. `run` steps MAY define an optional `cwd` field, interpreted as a relative path inside the sandbox root. If omitted, `cwd` defaults to the sandbox root. The runner MUST: - interpret `run` as a single command invocation (no newline; no shell operators) - parse `run` into argv using deterministic POSIX-style shellwords tokenization (single/double quotes and backslash escapes; unmatched quotes MUST fail) - reject any parsed argv that contains sh"
  r9,Legacy Playbook Format Is Rejected with Guidance,"The runner MUST reject legacy `llman x sdd-eval` playbooks that use the previous fixed-pipeline schema (for example, top-level `version: 1`). The error MUST be actionable and MUST mention that: - the playbook format has been replaced by a workflow/jobs/steps DSL, and - users must update the YAML to the new format."
scenarios[18]{req_id,id,given,when,then}:
  r1,"missing-workflow-fails-loudly","","a user runs `llman x sdd-eval run --playbook <path>` with a playbook missing `workflow.jobs`","the command exits non-zero"
  r1,"empty-job-steps-are-rejected","","a playbook job defines `steps: []`",playbook validation fails
  r1,"unknown-job-keys-are-rejected","","a playbook job defines an unknown key (for example `timeout:`)",playbook validation fails
  r2,"matrix-references-an-unknown-variant","","a playbook job defines `strategy.matrix.variant: [\"a\"]`","the command exits non-zero"
  r2,"variant-id-is-rejected-when-unsafe-for-paths","",a user defines a variant id containing `/` or `..`,playbook validation fails
  r2,"matrix-variant-expansion-is-serial-and-ordered","","a job defines `strategy.matrix.variant: [\"b\", \"a\"]`",the runner executes the job twice
  r3,"needs-references-an-unknown-job","","a job declares `needs: [\"missing\"]`",the playbook fails validation
  r3,"needs-cycle-is-rejected","",the workflow contains a dependency cycle (direct or indirect),the playbook fails validation
  r4,"step-with-both-uses-and-run-is-rejected","",a step defines both `uses` and `run`,playbook validation fails
  r4,"run-step-with-with-is-rejected","","a step defines `run: \"rg foo\"` and also defines `with: {}`",playbook validation fails
  r5,"unknown-built-in-action-fails-loudly","","a step uses `builtin:sdd-eval/does-not-exist`","the command exits non-zero"
  r6,"workspace-prepare-skips-secret-ish-files","",the project root contains `.env` or `.netrc`,those files are not present in the variant workspace
  r6,"acp-loop-action-writes-expected-artifacts","","the workflow runs `builtin:sdd-eval/acp.sdd-loop` for a variant","`acp-session.jsonl` exists under the variant `logs/`"
  r7,"interpolation-replaces-matrix-variant-in-run-steps","","a matrix-expanded job step uses `run: \"echo ${{ matrix.variant }}\"`",the runner interpolates the string before execution
  r7,"unknown-interpolation-path-fails-loudly","","a step string contains `${{ does.not.exist }}`","the command exits non-zero"
  r8,"disallowed-command-is-rejected","","a `run` step requests executing a non-allowlisted command (for example, `curl`)","the runner exits non-zero"
  r8,"cwd-path-traversal-is-rejected","","a `run` step sets `cwd: \"../outside\"`","the runner exits non-zero"
  r9,"legacy-version-1-playbook-fails-with-an-actionable-error","","a user runs `llman x sdd-eval run` with a playbook containing `version: 1`","the command exits non-zero"
