# Browsing

**Lightweight, headless-first MCP/API for browser automation**

A concise MCP server and Rust library: **navigate**, **get_links**, **follow_link**, **list_content** (links+images), **get_content**, **get_image**, **save_content**, **screenshot** (full or element). Lazy browser init. Parallel reads via RwLock. Runs headless by default with 17+ lightweight Chrome flags for fast, minimal operation. No LLM or AI dependencies.

## 🎯 Usage Modes

1. **🔌 MCP Server** (primary) - `navigate`, `get_links`, `follow_link`, `list_content`, `get_content`, `get_image`, `save_content`, `screenshot`, `generate_sitemap` tools
2. **⌨️ CLI** - Browser automation tasks
3. **📦 Library** - Browser automation via CDP

## ✨ Why Browsing?

Browser automation is challenging. You need to:

- **Extract structured data from unstructured HTML** - Parse complex DOM trees into usable representations
- **Handle browser automation reliably** - Manage browser lifecycle, CDP connections, and process management
- **Maintain testability** - Mock components for unit testing without real browsers
- **Support extensibility** - Add custom actions and browser backends

**Browsing solves all of this** with a clean, modular, and well-tested architecture.

## 🧠 Why Better Than Chrome DevTools?

Chrome DevTools Protocol (CDP) is an **engine** — it gives you raw browser control, but you must build all intelligence yourself. Most "headless browsing" tools are just CDP wrappers with slightly nicer APIs.

Browsing is a **self-driving car** on top of that engine. It adds five intelligence layers that are hard to replicate by simply wrapping CDP:

| Layer | CDP (Raw Protocol) | Browsing (Browser Automation) |
|---|---|---|
| **Perception** | Raw DOM tree | Semantic understanding: page intent, form schemas, action affordances |
| **Action** | Brittle `backend_node_id`/XPath | Self-healing element resolution: index → text → semantic fallback |
| **Memory** | Nothing persists | Session history, site-specific strategies |
| **Observability** | Protocol logs | DOM state snapshots and action traces |

**The goal**: CDP becomes invisible. Users think about tasks, not WebSocket messages or element IDs.

## 🎯 Key Features

### 🏗️ Trait-Based Architecture
- **BrowserClient trait** - Abstract browser operations for easy mocking and alternative backends
- **DOMProcessor trait** - Pluggable DOM processing implementations
- **ActionHandler trait** - Extensible action system for custom behaviors

### 🌐 Full Browser Automation
- **Headless by default** — `BrowserProfile::default()` runs headless with `--headless=new`
- **17+ lightweight flags** — disables extensions, sync, background networking, logging, and more
- Cross-platform support (macOS, Linux, Windows)
- Automatic browser detection
- Chrome DevTools Protocol (CDP) integration
- Tab management (create, switch, close)
- Screenshot capture (page and element-level)

### 📊 Advanced DOM Processing
- Full CDP integration (DOM, AX tree, Snapshot)
- Text serialization with interactive element indices
- Accessibility tree support for better semantic understanding

### 🔧 Extensible & Maintainable
- Manager-based architecture (TabManager, NavigationManager, ScreenshotManager)
- Custom action registration
- Utility traits for reduced code duplication
- Comprehensive test coverage (350+ tests)

## 📦 Installation

### As a Library

```toml
[dependencies]
browsing = "0.1"
tokio = { version = "1.40", features = ["full"] }
```

### As a CLI Tool

```bash
cargo install --path . --bin browsing
```

### As an MCP Server

```bash
cargo build --release --bin browsing-mcp
```

## 🚀 � Quick Start## � Quick Start

### 1️⃣ CLI Usage

```bash
# Run a browser automation task (headless by default)
browsing run "Navigate to example.com and extract content" --url https://news.ycombinator.com

# Run with visible browser
browsing run "Navigate to example.com and extract content" --url https://news.ycombinator.com --no-headless

# Launch a headless browser and get CDP URL (headless by default)
browsing launch

# Launch with visible browser
browsing launch --no-headless

# Connect to existing browser
browsing connect ws://localhost:9222/devtools/browser/abc123
```

**📖 [Full CLI Documentation](docs/CLI_USAGE.md)**

### 2️⃣ MCP Server Usage

Configure in Claude Desktop (`~/Library/Application Support/Claude/claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "browsing": {
      "command": "/path/to/browsing/target/release/browsing-mcp",
      "env": {
        "BROWSER_USE_HEADLESS": "true"
      }
    }
  }
}
```

Then ask Claude:
```
"Navigate to rust-lang.org, get the links, follow the second link, and screenshot the main content area"
```

**📖 [Full MCP Documentation](docs/MCP_USAGE.md)**

### 3️⃣ Library Usage

```rust
use anyhow::Result;
use browsing::Browser;
use browsing::browser::BrowserProfile;

#[tokio::main]
async fn main() -> Result<()> {
    browsing::init();

    let mut browser = Browser::new(BrowserProfile::default());
    browser.start().await?;

    browser.navigate("https://example.com").await?;

    let title = browser.get_current_page_title().await?;
    println!("Title: {}", title);

    let _ = browser.stop().await;
    Ok(())
}
```

**📖 [Full Library Documentation](docs/LIBRARY_USAGE.md)**

### Browser Launch Options

```rust
use browsing::{Browser, BrowserProfile};

// Option 1: Auto-launch headless browser (default)
let profile = BrowserProfile::default(); // headless = true by default
let browser = Browser::new(profile);

// Option 2: Launch with visible browser
let profile = BrowserProfile::default().with_headless(false);
let browser = Browser::new(profile);

// Option 3: Connect to existing browser
let browser = Browser::new(BrowserProfile::default())
    .with_cdp_url("http://localhost:9222".to_string());

// Option 4: Custom browser executable
use browsing::browser::launcher::BrowserLauncher;
let launcher = BrowserLauncher::new(profile)
    .with_executable_path(std::path::PathBuf::from("/path/to/chrome"));
```

### Using Traits for Testing

#### Standalone Browser Testing (no AI required)

```rust
use browsing::traits::{BrowserClient, DOMProcessor};
use std::sync::Arc;

// Test your browser automation logic directly
```


## 📚 Usage Examples

### Content Download

```rust
use browsing::{Browser, BrowserProfile};
use browsing::dom::DOMProcessorImpl;
use browsing::traits::DOMProcessor;

#[tokio::main]
async fn main() -> browsing::error::Result<()> {
    let mut browser = Browser::new(BrowserProfile::default());
    browser.start().await?;

    // Navigate to website
    browser.navigate("https://www.ibm.com").await?;
    tokio::time::sleep(tokio::time::Duration::from_secs(3)).await;

    // Extract content
    let cdp_client = browser.get_cdp_client()?;
    let session_id = browser.get_session_id()?;
    let target_id = browser.get_current_target_id()?;

    let dom_processor = DOMProcessorImpl::new()
        .with_cdp_client(cdp_client, session_id)
        .with_target_id(target_id);

    let state = dom_processor.get_serialized_dom().await?;
    let page_content = state.text_representation(None).unwrap_or_default();
    println!("Extracted {} bytes of content", page_content.len());

    // Save to file
    std::fs::write("ibm_content.txt", page_content)?;
    Ok(())
}
```

**Run this example:**
```bash
cargo run --example ibm_content_download
```

### Screenshot Capture

```rust
use browsing::Browser;

let browser = Browser::new(BrowserProfile::default());
browser.start().await?;

// Full page screenshot
let screenshot_data = browser.take_screenshot(
    Some("screenshot.png"),  // path
    true,                     // full_page
    None,                     // format
    None,                     // quality
).await?;

// Viewport only
let viewport = browser.take_screenshot(
    Some("viewport.png"),
    false,
    None,
    None,
).await?;
```

### Direct Browser Control

```rust
use browsing::{Browser, BrowserProfile};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut browser = Browser::new(BrowserProfile::default());
    browser.start().await?;

    // Navigate
    browser.navigate("https://example.com").await?;

    // Get current URL
    let url = browser.get_current_url().await?;
    println!("Current URL: {}", url);

    // Tab management
    use browsing::traits::BrowserClient;
    browser.create_tab(Some("https://hackernews.com")).await?;
    let tabs = browser.get_tabs().await?;
    println!("Open tabs: {}", tabs.len());

    // Switch tabs
    browser.switch_to_tab(&tabs[0].target_id).await?;

    Ok(())
}
```

### Custom Actions

```rust
use browsing::tools::views::{ActionHandler, ActionParams, ActionContext, ActionResult};
use browsing::error::Result;
use browsing::tools::Tools;

struct CustomActionHandler;

#[async_trait::async_trait]
impl ActionHandler for CustomActionHandler {
    async fn execute(
        &self,
        params: &ActionParams,
        context: &mut ActionContext<'_>,
    ) -> Result<ActionResult> {
        // Custom action logic here
        Ok(ActionResult {
            extracted_content: Some("Custom result".to_string()),
            ..Default::default()
        })
    }
}

// Register custom action
let mut tools = Tools::default();
tools.register_custom_action(
    "custom_action".to_string(),
    "Description of custom action".to_string(),
    None,  // domains
    CustomActionHandler,
);
```

## 🏗️ Architecture

Browsing follows **SOLID principles** with a focus on separation of concerns, testability, and maintainability.

```
┌─────────────────────────────────────────────────────────────┐
│                    Browser Automation                       │
│  ┌─────────────┬──────────────┬─────────────────────────┐  │
│  │   Browser   │ DOMProcessor │        Tools            │  │
│  │   (trait)   │    (trait)   │                         │  │
│  └──────┬──────┴──────┬───────┴────────────────┬────────┘  │
│         │             │                      │           │
└─────────┼─────────────┼──────────────────────┼───────────┘
          │             │                      │
    ┌─────▼──────┐ ┌───▼────┐        ┌────────▼─────┐
    │  Browser   │ │DomSvc  │        │   Handlers   │
    │            │ │        │        │              │
    │TabManager  │ │CDP     │        │Navigation    │
    │NavManager  │ │HTML    │        │Interaction   │
    │Screenshot  │ │Tree    │        │Tabs          │
    │            │ │Builder │        │Content       │
    └────────────┘ └────────┘        └──────────────┘
```

### Key Components

| Component | Responsibility | Trait-Based |
|-----------|---------------|-------------|
| **Browser** | Manages browser session and lifecycle | Implements `BrowserClient` |
| **DOMProcessor** | Extracts and serializes DOM | Implements `DOMProcessor` |
| **Tools** | Action registry and execution | Uses `BrowserClient` trait |
| **Handlers** | Specific action implementations | Use `ActionHandler` trait |

## 📁 Project Structure

```
browsing/
├── src/
│   ├── agent/              # Agent view types (used by tools layer)
│   │   ├── views.rs        # Data types
│   │   └── memory.rs       # Session memory types
│   ├── browser/            # Browser management
│   │   ├── session.rs      # Browser session (BrowserClient impl)
│   │   ├── tab_manager.rs  # Tab operations
│   │   ├── navigation.rs   # Navigation operations
│   │   ├── screenshot.rs   # Screenshot operations
│   │   ├── cdp.rs          # CDP WebSocket client
│   │   ├── launcher.rs     # Browser launcher
│   │   └── profile.rs      # Browser configuration
│   ├── dom/                # DOM processing
│   │   ├── processor.rs    # DOMProcessor trait impl
│   │   ├── serializer.rs   # Text serialization
│   │   ├── tree_builder.rs # DOM tree construction
│   │   ├── cdp_client.rs   # CDP wrapper for DOM
│   │   └── html_converter.rs # HTML to markdown
│   ├── tools/              # Action system
│   │   ├── service.rs      # Tools registry
│   │   ├── handlers/       # Action handlers
│   │   │   ├── navigation.rs
│   │   │   ├── interaction.rs
│   │   │   ├── tabs.rs
│   │   │   ├── content.rs
│   │   │   └── advanced.rs
│   │   └── params.rs       # Parameter extraction
│   ├── traits/             # Core trait abstractions
│   │   ├── browser_client.rs  # BrowserClient trait
│   │   └── dom_processor.rs   # DOMProcessor trait
│   ├── actor/              # Low-level interactions
│   │   ├── page.rs         # Page operations
│   │   ├── element.rs      # Element operations
│   │   ├── mouse.rs        # Mouse interactions
│   │   └── keyboard.rs     # Keyboard input
│   ├── config.rs           # Configuration
│   ├── error.rs            # Error types
│   ├── logging.rs          # Logging setup
│   ├── metrics.rs          # Metrics collection
│   ├── utils.rs            # Utilities
│   └── views.rs            # Shared data types
└── Cargo.toml
```

## 🎨 Design Principles

### Trait-Facing Design
- **BrowserClient** - Abstract browser operations for testing and alternative backends
- **DOMProcessor** - Pluggable DOM processing implementations
- **ActionHandler** - Extensible action system

### Separation of Concerns
- **TabManager** - Tab operations (create, switch, close)
- **NavigationManager** - Navigation logic
- **ScreenshotManager** - Screenshot capture
- **Handlers** - Focused action implementations

### DRY (Don't Repeat Yourself)
- **ActionParams** - Reusable parameter extraction
- **JSONExtractor** - Centralized JSON parsing
- **SessionGuard** - Unified session access

### KISS (Keep It Simple, Stupid)
- Split complex methods into focused helpers
- Clear naming and single responsibility
- Minimal dependencies between modules

## 🧪 Testing

```bash
# Run all tests
cargo test

# Run with output
cargo test -- --nocapture

# Run specific test
cargo test test_browser_navigation

# Run integration tests only
cargo test --test integration
```

### Test Coverage
- **350+ tests** across all modules (all passing, 0 ignored)
- **Test files**:
  - [actor_test.rs](tests/actor_test.rs) - Page, Element, Mouse, Keyboard operations
  - [browser_lifecycle_test.rs](tests/browser_lifecycle_test.rs) - Browser start/stop lifecycle
  - [browser_managers_test.rs](tests/browser_managers_test.rs) - Navigation, Screenshot, Tab managers
  - [browser_test.rs](tests/browser_test.rs) - Browser integration
  - [cdp_enhancements_test.rs](tests/cdp_enhancements_test.rs) - CDP client features
  - [dom_extraction_test.rs](tests/dom_extraction_test.rs) - DOM extraction
  - [dom_test.rs](tests/dom_test.rs) - DOM state and serialization
  - [error_handling_test.rs](tests/error_handling_test.rs) - Error variants and propagation
  - [error_recovery_test.rs](tests/error_recovery_test.rs) - Config validation and recovery
  - [integration_test.rs](tests/integration_test.rs) - Action models, URL extraction, tools
  - [integration_workflow_test.rs](tests/integration_workflow_test.rs) - Full workflow
  - [security_test.rs](tests/security_test.rs) - Security validation
  - [tools_handlers_test.rs](tests/tools_handlers_test.rs) - All action handlers
  - [tools_test.rs](tests/tools_test.rs) - Tools registry
  - [traits_test.rs](tests/traits_test.rs) - BrowserClient, DOMProcessor traits
  - [utils_test.rs](tests/utils_test.rs) - URL extraction, signal handling
  - [agent_test.rs](tests/agent_test.rs) - Agent view types
  - [agent_service_test.rs](tests/agent_service_test.rs) - Agent state and history
- **Mock implementations** for deterministic testing
- **Trait-based mocking** for browser/DOM components

## ⚠️ Data Retention Policy

### Browser Data is NEVER Deleted

**IMPORTANT**: The `browsing` library **never deletes browser data** for safety reasons.

#### What This Means:

| Data Type | Behavior |
|-----------|----------|
| **Bookmarks** | Never deleted |
| **History** | Never deleted |
| **Cookies** | Never deleted |
| **Passwords** | Never deleted |
| **Extensions** | Never deleted |
| **Cache** | Never deleted |
| **Temp Directories** | Never deleted (left in `/tmp/`) |

#### Why This Policy Exists:

1. **User Safety**: Users may specify a custom `user_data_dir` pointing to their real browser profile
2. **Catastrophe Prevention**: Accidentally deleting a user's real browser data (bookmarks, history, passwords) would be devastating
3. **Debugging**: Leaving temp directories allows inspection after crashes or failures
4. **User Control**: Users are responsible for managing their own browser data

#### How It Works:

When no `user_data_dir` is specified:
```rust
let profile = BrowserProfile {
    user_data_dir: None,  // Uses temp directory: /tmp/browser-use-1738369200000/
    ..Default::default()
};
```

When `browser.stop()` is called:
- ✅ Browser process is killed
- ✅ In-memory state is cleared
- ❌ User data directory is **NOT** deleted

#### Managing Temporary Data:

Users are responsible for cleanup:

```bash
# List browser temp directories
ls -la /tmp/browser-use-*

# Delete old temp directories (optional, manual cleanup)
rm -rf /tmp/browser-use-1738369200000/
```

#### Using a Custom Data Directory:

```rust
let profile = BrowserProfile {
    user_data_dir: Some("/path/to/custom/profile".into()),
    ..Default::default()
};
```

**Warning**: If you point to your real browser profile, the library will NOT protect it. You're responsible for that directory.



## 🔧 Configuration

### Browser Profile

```rust
use browsing::BrowserProfile;

// Default: headless mode with lightweight flags
let profile = BrowserProfile::default();

// Explicit headless
let profile = BrowserProfile::default().with_headless(true);

// Visible browser for debugging
let profile = BrowserProfile::default().with_headless(false);

// With custom data directory
let profile = BrowserProfile {
    user_data_dir: Some("/path/to/profile".into()),
    ..Default::default()
};
```


## 📖 API Documentation

Generate and view API docs:

```bash
cargo doc --open
```
