---
name: Provider backlog, S3 screenshots, DB scrapers (deduplicated)
overview: |
  Same product goals with minimal surface area: one scraper shape (ScraperConfig everywhere),
  one Athena admin guard, one DB cache sync used by boot/HTTP/CLI, one AWS S3 client stack,
  one headless stack for screenshots, and templating in a single small module (body context only in v1).
todos:
  - id: sql-ddl
    content: provision.sql — data_source_provider_requests + scraper_overrides (name UNIQUE, enabled, config JSONB as full ScraperConfig, updated_at); indexes; header comments listing Rust modules
    status: completed
  - id: api-athena-guard
    content: src/api/athena_guard.rs — shared 503 JSON + helper to borrow Athena client from AppState (eliminate copy-paste no_athena in each module)
    status: completed
  - id: data-layer
    content: src/data/provider_requests.rs (Athena CRUD as today); src/data/scraper_overrides.rs — list_enabled, optional upsert/delete if API needs it; map JSONB → ScraperConfig via serde (no parallel struct)
    status: completed
  - id: cache-sync
    content: Single async fn sync_scraper_overrides(athena, cache) in scraper_overrides; AppState.scraper_overrides_cache RwLock<HashMap<String, ScraperConfig>>; run_server, POST /api/scraper-overrides/reload, and CLI --prefetch-scraper-overrides all call the same fn
    status: completed
  - id: template-and-scrape
    content: src/scrape/template.rs — expand {{key}} and {{$TABLE.COL$}} (normalize case) for URL + header values; v1 context = flatten ScrapeRequest only (optional context field); defer Athena row hydration to a later milestone; resolve_scraper_config(name) — overrides cache then YAML; run_scrape uses expanded base_url + headers
    status: completed
  - id: api-routes
    content: provider_requests routes (503 via guard); scraper_overrides — GET list (masked if secrets), POST reload, optional POST/PUT one JSON body ScraperConfig + DELETE by name (thin handlers); POST /api/screenshot — optional provider_request_id patch
    status: completed
  - id: s3-screenshot
    content: Cargo — aws-config + aws-sdk-s3 only (rustls via default features; no second AWS stack); src/screenshot/ — S3 put + URL allowlist; chromiumoxide only for capture (async-aligned with tokio)
    status: completed
  - id: docs-env
    content: README — override priority, reload curl, AWS/S3/Chrome env, --prefetch-scraper-overrides
    status: completed
isProject: false
---

# Deduplicated design (complexity and SDK drift reduced)

## What changed vs the previous plan

- **One scraper model**: DB table stores a **single JSONB document** per row that deserializes to existing [`ScraperConfig`](src/config/scraper.rs). No `ModularScraperRow`, no parallel column set (`url_template` + duplicated `request`/`selectors` columns). **Templates live in `base_url` and header strings** exactly like YAML would express them.
- **One refresh primitive**: `sync_scraper_overrides` (name TBD) is the **only** place that queries Athena and fills `scraper_overrides_cache`. Boot, `POST .../reload`, and **`--prefetch-scraper-overrides`** call it—no second loading path.
- **One Athena API guard**: [`src/api/athena_guard.rs`](src/api/athena_guard.rs) holds the shared 503 payload and `Option<&AthenaClient>` extraction. [`http_profiles.rs`](src/api/http_profiles.rs) should be refactored to use it when touching this work (small mechanical change).
- **One AWS stack**: Add **`aws-config` + `aws-sdk-s3` only** in [`Cargo.toml`](Cargo.toml)—no overlapping legacy AWS crates.
- **One browser dependency**: **chromiumoxide** only for screenshots (async); do not add a second headless crate.
- **Templating**: Single module [`src/scrape/template.rs`](src/scrape/template.rs) used for **both** URL and header values. **v1** = context from request body only (add optional `context` or flatten fields on [`ScrapeRequest`](src/api/scrape.rs)). **Athena-backed `USERS.*` resolution** is explicitly **out of scope for first ship** (document as follow-up to avoid two code paths in v1).

## A. Provider backlog + screenshot (unchanged intent, leaner wiring)

- Table **`data_source_provider_requests`**: same fields as before (example URL, notes, status, optional screenshot key/id).
- **S3 upload**: configure from env in one place; construct **one** S3 client (or `aws_config::load_defaults` per process) reused by the screenshot handler.
- **`POST /api/screenshot`**: validate URL (SSRF-minded), render with chromiumoxide, `PutObject`, optionally update provider row.
- **Security/ops**: keep concurrency limit and bucket/prefix from env as a single config struct next to the S3 client builder (avoid scattering `var("AWS_...")`).

## B. DB scraper overrides (replaces “wide” scraper_configurations)

### Table `scraper_overrides` in [`sql/provision.sql`](sql/provision.sql)

| Column | Purpose |
|--------|--------|
| `id` | BIGSERIAL PK |
| `name` | TEXT **UNIQUE** NOT NULL — same as YAML/`config_name` |
| `enabled` | BOOL NOT NULL DEFAULT true |
| `config` | **JSONB NOT NULL** — serialized [`ScraperConfig`](src/config/scraper.rs) (same shape as a YAML file) |
| `updated_at` | TIMESTAMPTZ |

Index: unique(`name`), index on `enabled`, `updated_at`.

**Resolution order** (document in README): if `name` exists in **`scraper_overrides_cache`** and `enabled`, use DB config; else use [`scraper_configs`](src/lib.rs) from YAML.

### API surface (minimal duplication with http_profiles)

- **Read-only list** + **reload** are required for operations.
- **Optional CRUD**: prefer **one** `PUT /api/scraper-overrides/{name}` with body = full `ScraperConfig` JSON (same as YAML document) instead of field-by-field endpoints—reduces request types and validation drift.
- All mutating handlers call `sync_scraper_overrides` after success (or only reload if mutations are SQL-only in your workflow).

### Scrape handler

1. `resolve_config(name)` → DB override or YAML `ScraperConfig`.
2. Build template context from request (v1).
3. Expand `base_url` → final URL; expand header values.
4. Existing fetch + selector pipeline unchanged.

Backward compatibility: YAML configs keep requiring `url` in the request **unless** the resolved config uses a template-only URL; document rule (e.g. DB override with templated `base_url` may ignore explicit `url` or treat it as override—pick **one** rule and test).

## C. Architecture (single sync + single resolve)

```mermaid
flowchart TB
  subgraph sources [Sources]
    YAML[YAML files]
    ATH[(Athena scraper_overrides)]
  end
  subgraph once [Single primitives]
    Sync[sync_scraper_overrides]
    Resolve[resolve_scraper_config]
    Tpl[expand_template]
  end
  ATH --> Sync
  Sync --> Cache[RwLock overrides map]
  YAML --> Resolve
  Cache --> Resolve
  Resolve --> Tpl
  Tpl --> Scrape[Existing HTTP + selectors]
```

## D. Risks (unchanged reality, narrower scope)

- **chromiumoxide** ops weight remains; mitigated by documenting Chrome path and keeping screenshot path optional at runtime if env missing.
- **Large JSONB documents**: acceptable tradeoff for zero mapping layer; validates with `ScraperConfig` schema on write.

## E. Verification checklist

- Seed one override row whose `config.base_url` contains `https://example.com/user/{{username}}`; POST scrape with `context` / body fields; assert expanded URL.
- `POST .../reload` updates map without restart; `--prefetch-scraper-overrides` exits 0 and logs count.
- Provider screenshot → S3 key stored; **only** `aws-sdk-s3` on the wire for AWS.
