# dexter-sentinel

A small **Rust** service that runs **YAML-defined scrapers** over HTTP: fetch a page, apply CSS selectors, and optionally persist results to **AWS Athena** (via `athena_rs`). It exposes **Prometheus** metrics and an optional **Redis**-backed URL registry alongside an in-memory bloom filter.

## Run

```bash
cargo run
```

Defaults: bind `0.0.0.0:8080`. Override with `HOST` and `PORT`.

Optional `.env` (loaded automatically if present):

| Variable         | Purpose                                          |
| ---------------- | ------------------------------------------------ |
| `ATHENA_URL`     | Athena API base URL for `athena_rs`              |
| `ATHENA_API_KEY` | API key for Athena                               |
| `REDIS_URL`      | Redis connection string for the URL set registry |
| `RUST_LOG`       | e.g. `info`, `debug`                             |

## Config

Scraper definitions live under `config/scrapers/*.yaml`. See `config/scrapers/example.yaml` for shape (URL template, headers, selectors, output table name).

## HTTP API

- `GET /health` — liveness
- `GET /metrics` — Prometheus
- `POST /scrape` — body: JSON with `url` and `config_name`
- `GET /scrape/{config_name}?url=...` — scrape with named config
- `POST /api/cache/registry/add` — register a URL (bloom + Redis + Athena when configured)
- `GET /api/cache/registry/check?url=...` — check registry (Redis, then bloom)

## SQL

`sql/provision.sql` contains example PostgreSQL DDL aligned with default table/column names used by the app and example config.

## License

MIT — see `LICENSE`.
