# Specification: Browser Automation Library

## Vision

Browsing is a **standalone browser automation library** built on Chrome DevTools Protocol (CDP). It provides programmatic control over Chrome/Chromium for navigation, element interaction, content extraction, and screenshot capture — without any dependency on LLMs or AI services.

The core insight: **CDP is an engine. Browsing is the control panel.**

Most headless browsing tools are thin wrappers around CDP. They expose the protocol with a slightly nicer API but leave all the hard problems — interpreting pages, finding elements reliably, handling dynamic SPAs — to the developer. Browsing solves these problems directly with semantic DOM understanding and resilient element interaction.

## Design Principles

1. **CDP is invisible** — Users think about tasks, not WebSocket messages or `backend_node_id`
2. **Semantic over syntactic** — Understand what a page means, not just what its DOM contains
3. **Resilient over brittle** — Self-healing element resolution, not fragile selectors
4. **Stateful over stateless** — Session context and memory, not independent commands
5. **Observable over opaque** — DOM state snapshots and action traces for debugging

## Architecture Layers

### Layer 1: Perception — Semantic DOM Understanding

**Problem**: CDP gives you a raw DOM tree. Determining "this is a search form" from `<div class="x3f9"><input ...></div>` requires interpretation.

**Solution**: Enrich the DOM with inferred semantics before serialization.

- **Semantic role inference**: Classify elements by what they DO, not just what they ARE
  - `SearchForm`, `LoginForm`, `Navigation`, `ProductCard`, `Article`, `Pagination`, `FilterPanel`
- **Page intent classification**: On load, classify the page type
  - Login page, search results, product listing, product detail, article, error page, CAPTCHA
- **Form schema extraction**: Auto-detect form purpose and field metadata
  - "This is a search form with a query field and a category dropdown"
- **Action affordance tagging**: For each interactive element, tag what action it enables
  - "Submit form 'search'", "Navigate to /products", "Toggle filter panel"
- **Content hierarchy**: Distinguish main content, navigation, sidebar, ads, footer

**Implementation**: Extend `EnhancedDOMTreeNode` and `DOMInteractedElement` with `semantic_role` and `affordance` fields. Add a `PerceptionEngine` that runs as a post-processing pass after DOM tree construction.

**Success criteria**: DOM output includes semantic annotations that reduce ambiguity, without significantly increasing output size (achieved by pruning non-semantic elements).

### Layer 2: Action — Resilient Element Interaction

**Problem**: CDP relies on `backend_node_id` and XPath selectors that break constantly on React/Vue/Angular SPAs. Element indices shift, IDs are dynamic, and class names are hashed.

**Solution**: Multi-factor element resolution with automatic fallback.

- **Resolution chain**: When interacting with an element:
  1. **Index resolver** — Look up by index in `selector_map` (current approach)
  2. **Text match resolver** — Search by text content if index is stale
  3. **Semantic resolver** — "Find the submit button in the login form"
  4. **JavaScript resolver** — Execute JS to find by text/role/attribute
- **Self-healing selectors**: When an element is not found, automatically attempt the next resolver in the chain
- **Multi-factor addressing**: Combine text, position, visual features, and DOM path for robust identification

**Implementation**: Define an `ElementResolver` trait. Implement each strategy as a separate resolver. Chain them in a `FallbackResolver`.

**Success criteria**: Automation succeeds on dynamic SPAs where element indices shift between steps, with zero manual intervention.

### Layer 3: Memory — Persistent Session Context

**Problem**: Every session starts from zero. No memory of past site interactions, quirks, or progress.

**Solution**: Persistent memory layer.

- **Site-specific strategies**: Learn and remember how to interact with specific sites
  - "On example-airline.com, the search button is inside a shadow DOM"
- **Credential profiles**: Recognize login pages and auto-fill from known profiles
- **Progress tracking**: For long-running tasks, track what's been done and what's remaining
  - "Checked pages 1-3 of results, need to check page 4"
- **Session persistence**: Save and resume long-running tasks

**Implementation**: `SessionMemory` trait with a default `FileSystemMemory` implementation. Hook memory read/write into the session lifecycle.

**Success criteria**: Second visit to a site is measurably faster and more reliable than the first.

### Layer 4: Observability — Session Recording & Tracing

**Problem**: When automation fails, debugging means reading protocol logs. There's no easy way to "see" what the browser saw.

**Solution**: Capture DOM state and action traces for debugging.

- **DOM snapshots**: Capture serialized DOM state at each step
- **Action traces**: Record actions taken with element context and results
- **Screenshots** (optional): Capture visual state for visual debugging

**Success criteria**: When automation fails, a developer can inspect the DOM snapshot and action trace to understand the failure in under 2 minutes.

## Architecture

```
src/
├── perception/         # Semantic DOM enrichment, page classification
│   ├── engine.rs       # PerceptionEngine — runs post-DOM-build
│   ├── mod.rs          # Module exports
│   └── engine_test.rs  # Perception tests
├── browser/            # Browser lifecycle and CDP communication
│   ├── session.rs      # Browser session (BrowserClient impl)
│   ├── tab_manager.rs  # Tab operations
│   ├── navigation.rs   # Navigation operations
│   ├── screenshot.rs   # Screenshot operations
│   ├── cdp.rs          # CDP WebSocket client
│   ├── launcher.rs     # Browser launcher
│   ├── profile.rs      # Browser configuration
│   ├── views.rs        # Data types
│   └── mod.rs
├── dom/                # DOM processing
│   ├── processor.rs    # DOMProcessor trait impl
│   ├── serializer.rs   # Text serialization
│   ├── service.rs      # DOM service
│   ├── tree_builder.rs # DOM tree construction
│   ├── cdp_client.rs   # CDP wrapper for DOM
│   ├── html_converter.rs # HTML to markdown
│   ├── views.rs        # Data types
│   └── mod.rs
├── tools/              # Action system
│   ├── service.rs      # Tools registry
│   ├── handlers/       # Action handlers
│   │   ├── navigation.rs
│   │   ├── interaction.rs
│   │   ├── tabs.rs
│   │   ├── content.rs
│   │   ├── advanced.rs
│   │   └── mod.rs
│   ├── views.rs        # Data types
│   └── mod.rs
├── traits/             # Core trait abstractions
│   ├── browser_client.rs  # BrowserClient trait
│   ├── dom_processor.rs   # DOMProcessor trait
│   └── mod.rs
├── actor/              # Low-level interactions
│   ├── page.rs         # Page operations
│   ├── element.rs      # Element operations
│   ├── mouse.rs        # Mouse interactions
│   ├── keyboard.rs     # Keyboard input
│   └── mod.rs
├── config.rs           # Configuration
├── error.rs            # Error types
├── logging.rs          # Logging setup
├── metrics.rs          # Metrics collection
├── utils.rs            # Utilities
├── views.rs            # Shared data types
└── lib.rs              # Public API
```

## Phased Implementation

### Phase 1: Perception (Complete)
- Add `semantic_role` to `EnhancedDOMTreeNode` and `DOMInteractedElement`
- Implement `PerceptionEngine` with heuristic-based role inference
- Add page intent classification to DOM service
- Update serializer to include semantic annotations
- **Verify**: DOM output includes semantic tags; output size does not increase significantly

### Phase 2: Resilient Action (Complete)
- [x] Define `ElementResolver` with `ResolverConfig` and `ResolutionStrategy`
- [x] Implement 5-strategy fallback chain: Index → Selector → TextMatch → SemanticRole → JavaScript
- [x] Build `ElementResolver::resolve` with confidence scoring
- [x] Update action handlers to use resolver chain (`InteractionHandler`)
- [x] Add unit tests for resolver config, strategies, and selector map
- [x] **Verify**: `_verify_backend_node` uses real CDP `DOM.describeNode` instead of stub

### Phase 3: Memory (Complete)
- [x] Implement `AgentMemory` with `ProgressTracker`, `CredentialProfile`, `SiteStrategy`
- [x] Add `save_to_file` / `load_from_file` (excludes credentials for security)
- [x] Build `build_context` for domain-aware prompt enrichment
- [x] Wire `AgentMemory` into `ActionContext` so action handlers can access memory
- [x] **Verify**: Memory persistence round-trip works (progress + strategies saved/loaded)

### Phase 4: Observability (In Progress)
- [x] `SessionRecorder` — Records steps with timestamp, action type, params, URL, result, duration
- [x] JSON trace export — `SessionRecorder::export_to_file` writes `SessionTrace` with metadata
- [x] Configurable recording — `RecorderConfig` controls max steps, DOM capture, truncation
- [x] Automatic action recording — `Tools::act` records every action when a recorder is provided
- [ ] DOM snapshot capture at each step (pass via `dom_snapshot` param to `Tools::act`)
- [ ] **Verify**: A failed session produces a trace that explains the failure

## Success Metrics

| Metric | Target |
|---|---|
| Element resolution success on dynamic SPAs | >95% |
| Multi-step task completion without intervention | >85% |
| Time to debug a failed run | <2 minutes |
| Second-visit speedup vs first visit | >30% |

## Risk & Mitigation

| Risk | Mitigation |
|---|---|
| Output size increase from semantic annotations | Prune non-semantic elements aggressively; use hierarchical summarization |
| Complexity explosion from new modules | Implement one module at a time; each module is optional and can be disabled |
| Heuristic-based inference being wrong | Confidence scoring; fall back to raw DOM when confidence is low |
| Performance overhead of perception layer | Run perception asynchronously; cache results per page load |
