# ras-crawl

Reusable crawling, extraction, and optional GUI building blocks for Rust data collection tools.

`ras-crawl` is short for **Resource Acquisition and Structuring Crawler**. The crate handles the generic work of finding pages, turning HTML into extraction-friendly text, running extractors, saving checkpoints, and optionally wiring that flow into an `egui` GUI. Downstream projects provide their own structured `Info` type and runtime-specific orchestration.

## What It Provides

- `crawler`: HTTP crawling primitives, URL filtering, sitemap seeding, checkpoint support, and optional headless browser helpers.
- `info`: the `Info` trait for structured output models plus lightweight regex helpers and a built-in `Emails` type.
- `extractor`: a generic extraction trait and an optional OpenAI-backed extractor.
- `gui`: optional reusable GUI shell for crawl/extract runtimes.

## Installation

Use the default features for the core crawler, sitemap support, URL filtering, URL prioritization, query helpers, and checkpoints:

```toml
[dependencies]
ras-crawl = "0.1"
```

Enable optional integrations as needed:

```toml
[dependencies]
ras-crawl = {
    version = "0.1",
    features = ["gui", "headless", "openai"]
}
```

## Feature Flags

- `default`: enables `checkpoints`, `url-filtering`, `url-priority`, `query-helpers`, and `sitemap`.
- `checkpoints`: enables CBOR checkpoint save/load support.
- `url-filtering`: enables custom URL predicates in `CrawlerConfig`.
- `url-priority`: enables custom URL scoring in `CrawlerConfig` so higher-priority links are crawled sooner.
- `query-helpers`: enables reusable query-string helpers such as tracking-parameter filtering and allowlisted query-key checks.
- `sitemap`: seeds crawls from sitemap URLs before falling back to normal crawling.
- `headless`: enables Chromium-backed rendered-page fetching.
- `gui`: enables the reusable `egui` application shell.
- `openai`: enables `OpenAIExtractor`.

Use fewer features when embedding `ras-crawl` in libraries or CLIs that do not need GUI/headless/OpenAI dependencies.

## Basic Crawl

```rust,no_run
use ras_crawl::crawler::{Crawler, CrawlerConfig};
use ras_crawl::Crawl;
use url::Url;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let crawler = Crawler::new(CrawlerConfig {
        start_url: Some(Url::parse("https://example.com")?),
        max_pages: 10,
        ..Default::default()
    })?;

    let result = crawler.crawl().await?;

    for (url, html) in result.pages {
        println!("{url}: {} bytes", html.len());
    }

    Ok(())
}
```

With the `url-filtering` feature, restrict which links can be queued:

```rust,no_run
use ras_crawl::crawler::{Crawler, CrawlerConfig};
use url::Url;

let start = Url::parse("https://example.com/docs")?;
let host = start.host_str().map(str::to_string);

let crawler = Crawler::new(CrawlerConfig {
    start_url: Some(start),
    url_filter: Some(Box::new(move |url: &Url| url.host_str() == host.as_deref())),
    ..Default::default()
})?;
# Ok::<(), Box<dyn std::error::Error>>(())
```

With the `url-priority` feature, prefer some in-scope links over others:

```rust,no_run
use ras_crawl::crawler::{Crawler, CrawlerConfig};
use url::Url;

let crawler = Crawler::new(CrawlerConfig {
    start_url: Some(Url::parse("https://example.com/exhibitors")?),
    url_priority: Some(Box::new(|url: &Url| {
        if url.path().contains("exhibitor") {
            10
        } else {
            0
        }
    })),
    ..Default::default()
})?;
# Ok::<(), Box<dyn std::error::Error>>(())
```

With the `query-helpers` feature, downstream crates can build reusable filtering rules without hardcoding domain policy into `ras-crawl`:

```rust,no_run
use ras_crawl::crawler::utils::{keyword_score, query_keys_allowed};
use url::Url;

let url = Url::parse("https://example.com/exhibitors?page=2&utm_source=newsletter")?;

assert!(query_keys_allowed(&url, &["page"]));
assert_eq!(keyword_score(url.path(), &[("exhibitor", 10), ("speaker", -10)]), 10);
# Ok::<(), Box<dyn std::error::Error>>(())
```

The same feature also provides generic URL-pattern memory for downstream runtimes that want adaptive exploration without baking domain-specific "success" rules into `ras-crawl`:

```rust,no_run
use ras_crawl::crawler::utils::UrlPatternMemory;
use url::Url;

let mut memory = UrlPatternMemory::default();
memory.record_success(&Url::parse("https://example.com/exhibitors/acme?page=2")?);

let candidate = Url::parse("https://example.com/exhibitors/beta?page=3")?;
assert!(memory.best_similarity_score(&candidate) > 0);
# Ok::<(), Box<dyn std::error::Error>>(())
```

## Defining An `Info` Type

`Info` is the structured output contract used by extractors and save helpers. Implement it on your domain model, derive `serde`, `schemars`, and `validator`, then provide a JSON schema.

```rust,no_run
use ras_crawl::extractor::ExtractorError;
use ras_crawl::Info;
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use validator::Validate;

#[derive(Clone, Debug, Serialize, Deserialize, Validate, JsonSchema)]
#[schemars(
    title = "CompanyInfo",
    description = "Structured company record extracted from source pages."
)]
struct CompanyInfo {
    #[schemars(description = "Company display name.")]
    name: Option<String>,
    #[schemars(description = "Preferred company website URL.")]
    website: Option<String>,
}

impl Info for CompanyInfo {
    fn json_schema() -> serde_json::Value {
        serde_json::to_value(schemars::schema_for!(Self)).unwrap_or_default()
    }

    fn post_process(&mut self, original_text: &str) -> Result<(), ExtractorError> {
        if self.name.is_none() && original_text.contains("Example Ltd") {
            self.name = Some("Example Ltd".to_string());
        }
        Ok(())
    }
}
```

`post_process` is useful for deterministic cleanup after extraction: regex enrichment, deduping, URL normalization, or selecting preferred contact fields.

When using `OpenAIExtractor`, prefer expressing schema guidance through `schemars` metadata:

- Rust doc comments become schema descriptions when supported by `schemars`.
- `#[schemars(title = "...")]` sets the schema title used for the OpenAI response-format name.
- `#[schemars(description = "...")]` on structs and fields sets the structured-output descriptions the model sees.

`ras-crawl` also normalizes the derived schema for OpenAI Structured Outputs by filling strict-object requirements such as `additionalProperties: false` and `required`.

## OpenAI Extraction

Enable the `openai` feature and set `OPENAI_API_KEY` either at compile time or runtime:

```sh
OPENAI_API_KEY=sk-... cargo run
```

Create an extractor with the default model:

```rust,no_run
use ras_crawl::extractor::{Extract, OpenAIExtractor};
# use ras_crawl::Emails;

# async fn run() -> Result<(), Box<dyn std::error::Error>> {
let extractor = OpenAIExtractor::<Emails>::new()?;
let items = vec![(
    "example.com".to_string(),
    vec!["Contact us at hello@example.com".to_string()],
)];

let results = extractor.extract_all(&items).await?;
# Ok(())
# }
```

Or choose a model explicitly:

```rust,no_run
# use ras_crawl::extractor::OpenAIExtractor;
# use ras_crawl::Emails;
let extractor = OpenAIExtractor::<Emails>::with_model("gpt-5-nano")?;
# Ok::<(), Box<dyn std::error::Error>>(())
```

The OpenAI extractor batches input automatically, reserves output tokens, truncates oversized items, retries rate limits, and splits batches when model output is truncated. It derives the response-format name and description from your `schemars` schema metadata, so downstream `Info` types can steer extraction semantics without hand-writing JSON schema.

## Checkpoints

With `checkpoints` enabled, use `crawler::checkpoint::Checkpoint<T>` to persist:

- visited URLs,
- discovered target URLs,
- source listing text for each target,
- fetched page text for each target,
- extracted results.

Checkpoints are CBOR-encoded and intentionally have human-readable filenames:

```rust,no_run
use ras_crawl::crawler::checkpoint::Checkpoint;
use ras_crawl::Emails;
use url::Url;

let start = Url::parse("https://example.com")?;
let checkpoint = Checkpoint::<Emails>::new(start.clone());
let filename = Checkpoint::<Emails>::checkpoint_filename(&start);

checkpoint.save(std::path::Path::new(&filename))?;
# Ok::<(), Box<dyn std::error::Error>>(())
```

## GUI Runtime

Enable the `gui` feature to use the reusable `App<I, R>` shell. You supply:

- an `Info` type,
- a runtime implementing `gui::Runtime<I>`,
- a Tokio runtime handle,
- a crate name for log filtering.

```rust,no_run
# #[cfg(feature = "gui")]
# mod example {
use ras_crawl::gui::{GuiError, Runtime, RuntimeConfig};
use ras_crawl::{Emails, gui::App};
use tokio_util::sync::CancellationToken;

struct MyRuntime;

impl Runtime<Emails> for MyRuntime {
    async fn run(
        &self,
        config: RuntimeConfig,
        cancel: CancellationToken,
    ) -> Result<(), GuiError> {
        let _ = (config, cancel);
        Ok(())
    }

    fn render_config_extension(&self, ui: &mut egui::Ui, enabled: bool) {
        ui.add_enabled_ui(enabled, |ui| {
            ui.label("Custom runtime controls go here");
        });
    }

    fn extension_tabs(&self) -> &'static [&'static str] {
        &["Import"]
    }

    fn render_extension_tab(
        &self,
        tab: &str,
        ui: &mut egui::Ui,
        crawler_enabled: bool,
        handle: &tokio::runtime::Handle,
    ) {
        let _ = (tab, crawler_enabled, handle);
        ui.label("Custom tab content goes here");
    }
}

fn build_app(handle: tokio::runtime::Handle) -> Result<App<Emails, MyRuntime>, GuiError> {
    App::new(handle, "my_crawler", MyRuntime)
}
# }
```

`render_config_extension` adds controls to the crawler configuration panel. `extension_tabs` and `render_extension_tab` add runtime-owned tabs next to the default crawler tab for related workflows such as imports, exports, or CRM syncs. The GUI passes the Tokio handle into extension tabs so downstream runtimes can spawn async work without owning a second runtime.

The GUI is intended as a starting point. Downstream crates should inject their own runtime, export logic, tabs, and `Info` definitions rather than modifying the core crawler.

## Headless Fetching

Enable `headless` to use `crawler::headless::HeadlessBrowser` for JavaScript-rendered pages. It uses a system Chrome/Chromium installation through `chromiumoxide`.

Typical use is a second pass for targets where normal HTTP crawling missed contact information.

## Building From Source

```sh
cargo build
cargo test --all-features
cargo clippy --all-features --all-targets -- -D warnings
cargo fmt --check
```

For OpenAI-backed extraction, provide `OPENAI_API_KEY` before running code that constructs `OpenAIExtractor`.

## Current Scope

`ras-crawl` is a library crate. It provides reusable crawler/extractor/GUI pieces, not a finished domain crawler. Build application-specific behavior in downstream crates by defining an `Info` model and runtime orchestration.
