# dexter-sentinel

current version: `0.5.1`
A small **Rust** service that runs **YAML-defined scrapers** over HTTP: fetch a page, apply CSS selectors, and optionally persist results to **AWS Athena** (via `athena_rs`). It exposes **Prometheus** metrics and an optional **Redis**-backed URL registry alongside an in-memory bloom filter.

## Run

```bash
cargo run
```

Defaults: bind `0.0.0.0:8080`. Override with `HOST` and `PORT`.

Optional `.env` (loaded automatically if present):

| Variable                 | Purpose                                                                 |
| ------------------------ | ----------------------------------------------------------------------- |
| `ATHENA_URL`             | Athena API base URL for `athena_rs`                                     |
| `ATHENA_API_KEY`         | API key for Athena                                                      |
| `ATHENA_CLIENT`          | Value for `X-Athena-Client` header (default `dexter`)                  |
| `REDIS_URL`              | Redis connection string for the URL set registry                        |
| `RUST_LOG`               | e.g. `info`, `debug`                                                    |
| `S3_SCREENSHOT_BUCKET`   | S3 bucket for `POST /api/screenshot` (optional; omit to disable route)  |
| `S3_SCREENSHOT_PREFIX`   | Key prefix, default `screenshots/`                                        |
| `S3_PUBLICATIONS_BUCKET` | S3 bucket for `POST /api/publications/pwc-al/sync` and `pwc-al-publications` CLI (optional) |
| `S3_PUBLICATIONS_PREFIX` | Key prefix, default `publications/pwc-al/`                                |
| `S3_PUBLICATIONS_SKIP_EXISTING` | If unset or not `false`/`0`, skip upload when object already exists (`head_object`) |
| `SCREENSHOT_MAX_CONCURRENT` | Max parallel Chromium screenshot jobs (default `2`; `0` = unlimited) |
| `MAX_WORKER_THREADS`      | Actix HTTP worker threads (default auto; must be a positive integer)     |
| `AWS_REGION` / std creds | `aws_config::load_from_env` for S3                                        |
| `CHROME_EXECUTABLE`      | Path to Chromium/Chrome for screenshots (otherwise auto-detect)         |

Scraper config resolution: if Athena is configured, enabled rows in `scraper_overrides` **replace** the YAML entry for the same `name`. Reload without restart:

`curl -X POST http://localhost:8080/api/scraper-overrides/reload`

CLI smoke check (loads overrides then exits): `cargo run -- --prefetch-scraper-overrides`

## CLI binaries

Use `cargo run --bin <name> -- …` (note the `--` before flags). ATHENA uploads need `ATHENA_API_KEY` (or `--athena-key`) unless you pass `--no-insert`.

### `dexter-sentinel` (HTTP server)

```bash
cargo run
cargo run -- --host 0.0.0.0 --port 8080
cargo run -- --redis-url "$REDIS_URL" --athena-url "$ATHENA_URL" --athena-key "$ATHENA_API_KEY"
cargo run -- --athena-client "neon_dexter_sentinel"
cargo run -- --prefetch-scraper-overrides
```

### `dexter-scrape` — country × search engine

Rules enforced by the binary: **KVK** only with **NL**. **EGRUL** only with **RU**. **registries-io** only with **PL** or **CZ**. **Bizzy VAT** enrichment only with **NL** + **kvk**.

_Default search engine:_ KVK when `-c NL`, otherwise Northdata. PL/CZ without `-s` therefore use Northdata; add `-s registries-io` for registries.io.

**Single pass** (requires `-q` / `--query`):

```bash
# NL — KVK (default engine for NL)
cargo run --bin dexter-scrape -- -c NL -q "shell"

# NL — KVK explicit
cargo run --bin dexter-scrape -- -c NL -s kvk -q "shell"

# NL — KVK + Bizzy VAT backfill
cargo run --bin dexter-scrape -- -c NL -s kvk --vat-id-search-engine bizzy -q "shell"

# NL — Northdata (optional cookie file or NORTHDATA_COOKIE env)
cargo run --bin dexter-scrape -- -c NL -s northdata -q "shell"
cargo run --bin dexter-scrape -- -c NL -s northdata --northdata-cookie-file ./northdata_cookie.txt -q "shell"

# Non-NL — Northdata (default engine): Estonia example
cargo run --bin dexter-scrape -- -c EE -q "example"

# Non-NL — Northdata explicit
cargo run --bin dexter-scrape -- -c DE -s northdata -q "gmbh"

# RU — EGRUL only
cargo run --bin dexter-scrape -- -c RU -s egrul -q "ооо"

# PL — Northdata (default for non-NL)
cargo run --bin dexter-scrape -- -c PL -q "transport"

# PL — registries.io
cargo run --bin dexter-scrape -- -c PL -s registries-io -q "transport"

# CZ — Northdata (default)
cargo run --bin dexter-scrape -- -c CZ -q "praha"

# CZ — registries.io
cargo run --bin dexter-scrape -- -c CZ -s registries-io -q "praha"
```

**Continuous / randomized loop** (`-r` / `--randomize-query`; `-q` optional as seed for `query_seed`):

```bash
cargo run --bin dexter-scrape -- -c NL -r
cargo run --bin dexter-scrape -- -c NL -s northdata -r
cargo run --bin dexter-scrape -- -c NL -s kvk --vat-id-search-engine bizzy -r
cargo run --bin dexter-scrape -- -c EE -r
cargo run --bin dexter-scrape -- -c RU -s egrul -r
cargo run --bin dexter-scrape -- -c PL -r
cargo run --bin dexter-scrape -- -c PL -s registries-io -r
cargo run --bin dexter-scrape -- -c CZ -r
cargo run --bin dexter-scrape -- -c CZ -s registries-io -r
# Optional: custom word list / seed CSV / disable built-in CSV seeds
cargo run --bin dexter-scrape -- -c NL -r --randomize-words-file scripts/words.txt
cargo run --bin dexter-scrape -- -c NL -r --randomize-seed-csv ./names.csv
cargo run --bin dexter-scrape -- -c NL -r --randomize-no-seed-csv
```

**Dry run (no Athena), telemetry, CSV append, output path:**

```bash
cargo run --bin dexter-scrape -- -c PL -s registries-io -q "a" --no-insert
cargo run --bin dexter-scrape -- -c NL -q "a" --no-insert --telemetry
cargo run --bin dexter-scrape -- -c NL -q "a" -o ./out.csv --append-csv
```

**Athena / HTTP profile / proxy (insert mode):**

```bash
cargo run --bin dexter-scrape -- -c NL -q "a" \
  --athena-url https://mirror3.athena-db.com \
  --athena-key "$ATHENA_API_KEY" \
  --athena-client dexter
cargo run --bin dexter-scrape -- -c NL -q "a" --http-profile my_profile_name
cargo run --bin dexter-scrape -- -c NL -q "a" --proxy-url "http://127.0.0.1:8888"
```

**Useful overrides** (combine with any flow above):

```bash
cargo run --bin dexter-scrape -- -c NL -q "a" --min-query-length 4 --max-query-length 10
cargo run --bin dexter-scrape -- -c PL -s registries-io -q "a" --registries-io-base-url https://registries.io --registries-io-page-limit 50
cargo run --bin dexter-scrape -- -c RU -s egrul -q "a" --egrul-base-url https://egrul.nalog.ru
cargo run --bin dexter-scrape -- -c NL -q "a" --kvk-api-url https://web-api.kvk.nl/zoeken/v3/search
cargo run --bin dexter-scrape -- -c EE -q "a" --northdata-base-url https://www.northdata.com --max-workers 10
```

### `nl-scrape` (KVK → `nl_companies` + global registry)

NL-only; no country flag.

```bash
cargo run --bin nl-scrape -- -q "shell"
cargo run --bin nl-scrape -- -r
cargo run --bin nl-scrape -- -q "shell" --no-insert
cargo run --bin nl-scrape -- -q "shell" -o nl_out.csv
cargo run --bin nl-scrape -- -q "shell" --telemetry
cargo run --bin nl-scrape -- -q "shell" --max-workers 10 --max-query-length 8
cargo run --bin nl-scrape -- -q "shell" --http-profile my_profile_name
cargo run --bin nl-scrape -- -q "shell" --proxy-url "http://127.0.0.1:8888"
cargo run --bin nl-scrape -- -q "shell" \
  --athena-url https://cluster.athena-db.com \
  --athena-key "$ATHENA_API_KEY" \
  --athena-client dexter
```

### `pwc-al-publications` (PwC Albania publications → S3)

Fetches [PwC Albania — Publications](https://www.pwc.com/al/en/Publications.html), parses Tax / Legal listing links, downloads PDFs (and optional HTML pages), uploads under `S3_PUBLICATIONS_PREFIX` (`pdf/` and `html/`). Requires `S3_PUBLICATIONS_BUCKET` unless `--dry-run`.

```bash
cargo run --bin pwc-al-publications -- --dry-run
cargo run --bin pwc-al-publications --
cargo run --bin pwc-al-publications -- --include-html
```

Equivalent HTTP: `POST /api/publications/pwc-al/sync` with JSON `{ "dry_run": false, "include_html": false }`.

### `tax-harvest` (country/year tax obligations → `tax_types` + `tax_rates` + `tax_due_dates`)

Supports hybrid tax harvesting:
- `curated-json` mode (implemented): load normalized records from a JSON file.
- `web` mode (scaffold): fetch JSON from a URL and parse the same shape.

```bash
# Curated JSON dry run (no insert)
cargo run --bin tax-harvest -- \
  -c NL -y 2026 \
  --source-mode curated-json \
  --input ./tax-data/nl-2026.json \
  --no-insert

# Curated JSON insert mode
cargo run --bin tax-harvest -- \
  -c NL -y 2026 \
  --source-mode curated-json \
  --input ./tax-data/nl-2026.json \
  --athena-key "$ATHENA_API_KEY"

# Web mode scaffold (expects JSON endpoint)
cargo run --bin tax-harvest -- \
  -c PL -y 2026 \
  --source-mode web \
  --source-url https://example.test/poland-tax-2026.json \
  --no-insert
```

Curated JSON accepts either a top-level array or a wrapped object with `records`:

```json
{
  "country_code": "NL",
  "tax_year": 2026,
  "records": [
    {
      "tax_category": "CIT",
      "obligation_code": "cit_annual",
      "name": "Corporate income tax",
      "rates": [
        {
          "rate_kind": "flat",
          "rate_percent": 25.8,
          "effective_from": "2026-01-01"
        }
      ],
      "due_dates": [
        {
          "period_code": "annual",
          "due_date": "2027-05-31"
        }
      ]
    }
  ]
}
```

## Config

Scraper definitions live under `config/scrapers/*.yaml`. See `config/scrapers/example.yaml` for shape (same JSON shape as the `config` JSONB column in `scraper_overrides`). Optional `{{placeholders}}` in `base_url` and header values are filled from the optional `context` object on `POST /scrape`. If `url` is omitted on POST, `base_url` is used after expansion.

## HTTP API

- `GET /` — API root index (service status + quick endpoint map)
- `GET /health` — liveness
- `GET /metrics` — Prometheus (includes uptime, worker threads, and build info/version)
- `POST /scrape` — body: `config_name`, optional `url`, optional `context` (template map)
- `GET /scrape/{config_name}?url=...` — `url` optional if config `base_url` is enough (after expansion)
- `POST /api/cache/registry/add` — register a URL (bloom + Redis + Athena when configured)
- `GET /api/cache/registry/check?url=...` — check registry (Redis, then bloom)
- `GET|POST /api/provider-requests`, `GET|PUT|DELETE /api/provider-requests/{id}` — data-source backlog (503 if Athena unset)
- `GET /api/scraper-overrides`, `POST /api/scraper-overrides` (body: full scraper JSON), `POST .../reload`, `PUT|DELETE /api/scraper-overrides/{name}`
- `POST /api/screenshot` — body: `{ "url", "provider_request_id"? }` (S3 + optional backlog row update); URLs are restricted against obvious local/private targets
- `POST /api/publications/pwc-al/sync` — body: `{ "dry_run"?, "include_html"? }`; PwC Albania publications → S3 (`S3_PUBLICATIONS_BUCKET`); **503** if bucket unset and `dry_run` is false
- `GET|POST /api/http-profiles` — unchanged (uses shared Athena guard)

## SQL

`sql/provision.sql` contains PostgreSQL DDL aligned with default table/column names used by the app and CLI pipelines.

Tax harvesting tables:
- `tax_types`
- `tax_rates`
- `tax_due_dates`

Tax schema migration:
- `sql/migrations/20260407100000_tax_rates.sql`

## License

MIT — see `LICENSE`.



## Companies and legal entities

### Norway (NO)

For Norwegian organizations, prefer the official **Enhetsregisteret** (entity register) open API from the Brønnøysund Register Centre rather than scraping HTML: search and sync by organization number, filtering, and bulk-friendly JSON. Documentation: [Enhetsregisteret API — overview of services](https://data.brreg.no/enhetsregisteret/api/dokumentasjon/no/index.html#section/Oversikt-over-tjenester).


