# Instruction Source Priority

Tracking: MIK-2925 Phase 5

AXTerminator treats UI automation input as a many-source system. A caller may
provide a user request, agent-generated MCP tool arguments, AX API facts, screen
vision output, and app dialogs in the same workflow. When those sources disagree,
the lower-trust source must not silently override the higher-trust one.

## Precedence

### UI Facts

For the same field or target, AXTerminator uses this order:

1. `ax_api` - macOS Accessibility API facts.
2. `app_dialog` - first-party app dialog facts.
3. `screen_vision` - screenshot or VLM-inferred facts.

AX facts are preferred because they are structural, typed, and usually provided
by the target app or operating system. Screen vision is a fallback for surfaces
that do not expose useful accessibility structure, such as canvas, games, video,
and other renderer-only regions.

### Tool Intent

For agent-mediated MCP calls, AXTerminator uses this order:

1. `human_user_prompt` - the initiating user request.
2. `agent_tool_args` - tool arguments generated by an agent.

For direct human tool calls, `human_tool_args` are the user instruction. There is
no separate agent layer to override.

## `ax_find_visual`

`ax_find_visual` applies this priority when explicit mode is enabled:

1. Resolve the effective description from `description`, optional
   `user_prompt`, and optional `caller`.
2. Try the live AX tree with the effective description.
3. If AX returns a confident match, return that match and skip screen vision.
4. If AX cannot resolve and the client supports MCP sampling, return the
   screenshot plus sampling request for VLM location.

The behavior is controlled by `AXTERMINATOR_PRIORITY_MODE`:

| Value | Meaning |
| --- | --- |
| unset or `legacy` | Preserve historical visual lookup behavior. This is the rollback default. |
| `explicit` | Enforce the source-priority rules before screen-vision fallback. |

The optional MCP arguments are backward-compatible:

| Argument | Values | Default | Meaning |
| --- | --- | --- | --- |
| `caller` | `agent`, `human` | `agent` | Whether the tool args came from an agent or a direct human caller. |
| `user_prompt` | string | unset | Higher-priority initiating user prompt when an agent supplies tool args. |

Example agent-mediated call:

```json
{
  "app": "Preview",
  "description": "delete button",
  "caller": "agent",
  "user_prompt": "cancel the dialog"
}
```

The effective description is `cancel the dialog`, because the initiating user
prompt has higher priority than agent-supplied tool arguments.

## Conflict Examples

### AX API vs Screen Vision

If AX reports a button title as `Cancel` and screen vision infers the same button
as `Delete`, AXTerminator uses `Cancel`. The VLM result can still be logged or
shown for debugging, but it does not become the selected field value.

### App Dialog vs Screen Vision

If no AX fact exists and a first-party app dialog exposes a structured action
label, that dialog fact wins over a screenshot inference. This keeps system or
app-confirmation UI from being treated as ordinary pixels.

### Human Prompt vs Agent Tool Args

If an agent receives "cancel the dialog" but calls `ax_find_visual` with
`description="delete button"`, AXTerminator treats the user prompt as the
effective description when `caller="agent"` and `user_prompt` is supplied.

## Rationale

Many-tier instruction hierarchy research shows that real agents receive
conflicting instructions from far more sources than the traditional
system-user-tool model. AXTerminator's public surface includes user prompts, MCP
tool arguments, AX responses, screen content, and app dialogs, so it needs an
explicit order rather than relying on whichever source was processed last.
