# ChromaFrame vision helper

Local-only helper package for MediaPipe landmark extraction. Private images, overlays, virtual environments, cache directories, and model assets must stay outside this repository.

The Rust integration lives behind the `chromaframe-sdk` `vision-helper` feature and invokes this helper with temporary request/response JSON files. Helper stdout/stderr is never part of the public SDK or MCP output contract.

## Supported runtime

- Python 3.11 or 3.12
- Packages from `requirements.txt`: `mediapipe`, `opencv-python-headless`,
  `numpy`, `Pillow`, and `onnxruntime`
- A MediaPipe Face Landmarker `.task` file stored in a user cache or Downloads
  directory, never in the repo

## Rust environment variables

- `CHROMAFRAME_VISION_HELPER_PYTHON`: Python executable for the helper.
- `CHROMAFRAME_MEDIAPIPE_TASK_PATH`: local Face Landmarker `.task` model path.
- `CHROMAFRAME_FACE_PARSING_ONNX_PATH`: required baseline semantic parser model path.
- `CHROMAFRAME_VISION_ASSET_MANIFEST`: required parser manifest path.
- `CHROMAFRAME_VISION_HELPER_TIMEOUT_MS`: helper timeout, clamped by the SDK.

## Command contract

```sh
uv run python -m chromaframe_vision_helper extract --request-json /tmp/request.json --response-json /tmp/response.json
```

Protocol-level failures are written to `response.json` with `status: "error"`
and a typed `error_code`. The helper intentionally avoids writing raw stderr,
private paths, image metadata, or pixel payloads to protocol diagnostics.

## Model acquisition

Download the MediaPipe Face Landmarker task model to a local cache directory,
then point `CHROMAFRAME_MEDIAPIPE_TASK_PATH` at that file. Install the mandatory
baseline parser with:

```sh
uv run python sdk/vision-helper/scripts/setup-face-parser.py
```

The setup script downloads or copies `resnet18.onnx` to a local cache outside the
repository, verifies it against the checked-in expected SHA-256, and writes
`resnet18.manifest.json` only after the match. Runtime analysis never downloads
model assets. Do not commit model weights.

## Parser manifest

Provide a JSON manifest:

```json
{
  "face_parsing_onnx": {
    "manifest_schema_version": 1,
    "model_id": "PayamFard123/dermaintel-face-parsing/resnet18.onnx",
    "sha256": "<lowercase-hex-sha256>",
    "expected_sha256": "0d9bd318e46987c3bdbfacae9e2c0f461cae1c6ac6ea6d43bbe541a91727e33f",
    "license": "<approved license identifier>",
    "license_id": "<approved license identifier>",
    "approved_for_use": true,
    "model_type": "baseline_face_parser",
    "source_url": "https://huggingface.co/PayamFard123/dermaintel-face-parsing/resolve/c1c5fd399cc72b45e3613019879fc6c17a7d8f6a/resnet18.onnx",
    "source_revision": "c1c5fd399cc72b45e3613019879fc6c17a7d8f6a",
    "labels": {
      "background": 0,
      "skin": 1,
      "l_brow": 2,
      "r_brow": 3,
      "l_eye": 4,
      "r_eye": 5,
      "eye_g": 6,
      "l_ear": 7,
      "r_ear": 8,
      "ear_r": 9,
      "nose": 10,
      "mouth": 11,
      "u_lip": 12,
      "l_lip": 13,
      "neck": 14,
      "neck_l": 15,
      "cloth": 16,
      "hair": 17,
      "hat": 18
    },
    "input": { "width": 512, "height": 512 },
    "normalization": { "mean": [0.485, 0.456, 0.406], "std": [0.229, 0.224, 0.225] },
    "output": { "type": "logits_argmax" }
  }
}
```

Checksum mismatches and unapproved licenses are surfaced as typed SDK readiness
errors. Runtime consumes selected groups for scalp hair, skin, brow, and lip, but
the manifest must include the full exact 19-label map with exact key spelling and
IDs. Beard, moustache, stubble, and generic facial-hair labels are rejected for
baseline parsers; those require a future specialist model type.

## Overlay policy

Overlays are opt-in only. Callers provide an output path, preferably outside the
repository such as `~/Downloads`. When an overlay is requested, the helper also
writes a small `.json` sidecar with redacted path information and summary counts.
No overlay is generated by default.

The MCP server intentionally does not expose overlay generation; `chromaframe_analyze_image` is read-only and returns sanitized region/ranking summaries only.
