---
name: Dedup backlog scrapers S3
overview: "Consolidate the earlier plan: one ScraperConfig-shaped JSONB for DB overrides, shared Athena 503 guard, a single scraper sync function for boot/reload/CLI, aws-sdk-s3 + chromiumoxide only, and body-only templating in v1."
todos:
  - id: sql-ddl
    content: "provision.sql: data_source_provider_requests + scraper_overrides (name, enabled, config JSONB ScraperConfig, updated_at) + indexes + header comments"
    status: completed
  - id: api-athena-guard
    content: src/api/athena_guard.rs + refactor http_profiles to use shared 503/helper
    status: completed
  - id: data-layer
    content: data/provider_requests.rs + data/scraper_overrides.rs (list, serde JSONB→ScraperConfig, optional upsert/delete)
    status: completed
  - id: cache-sync
    content: sync_scraper_overrides + AppState cache; boot + reload + --prefetch-scraper-overrides call same fn
    status: completed
  - id: template-and-scrape
    content: scrape/template.rs + extend ScrapeRequest + resolve + run_scrape wiring (v1 body context only)
    status: completed
  - id: api-routes
    content: provider_requests + scraper_overrides + screenshot routes (thin handlers)
    status: completed
  - id: s3-screenshot
    content: aws-config + aws-sdk-s3 + chromiumoxide; src/screenshot/ URL allowlist + PutObject
    status: completed
  - id: docs-env
    content: README env and operational notes
    status: completed
isProject: false
---

# Deduplicated provider backlog, S3 screenshots, DB scrapers

## Core reductions

- **One scraper type end-to-end**: Table `scraper_overrides` with columns `name`, `enabled`, `config` **JSONB** (full [`ScraperConfig`](C:/Users/floris/Documents/GitHub/dexter-sentinel/src/config/scraper.rs)). No parallel `ModularScraperRow` / wide column set. Templates use existing `base_url` + header strings.
- **One sync function**: `sync_scraper_overrides` in [`src/data/scraper_overrides.rs`](C:/Users/floris/Documents/GitHub/dexter-sentinel/src/data/scraper_overrides.rs) — sole Athena→cache path. Called from [`run_server`](C:/Users/floris/Documents/GitHub/dexter-sentinel/src/lib.rs), `POST /api/scraper-overrides/reload`, and CLI `--prefetch-scraper-overrides`.
- **One API guard**: [`src/api/athena_guard.rs`](C:/Users/floris/Documents/GitHub/dexter-sentinel/src/api/athena_guard.rs) — shared 503 JSON + client borrow. Refactor [`src/api/http_profiles.rs`](C:/Users/floris/Documents/GitHub/dexter-sentinel/src/api/http_profiles.rs) to use it (mechanical).
- **SDK drift**: [`Cargo.toml`](C:/Users/floris/Documents/GitHub/dexter-sentinel/Cargo.toml) adds **`aws-config` + `aws-sdk-s3` only** (rustls). **chromiumoxide** only for screenshots — no second headless crate.
- **Templating**: Single [`src/scrape/template.rs`](C:/Users/floris/Documents/GitHub/dexter-sentinel/src/scrape/template.rs) for URL + headers. **v1** = request body / optional `context` only. **Defer** Athena-backed `USERS.*` hydration to a follow-up milestone.

## Resolution and scrape

- `resolve_scraper_config(name)`: `scraper_overrides_cache` (if enabled) else YAML map in `AppState`.
- [`run_scrape`](C:/Users/floris/Documents/GitHub/dexter-sentinel/src/api/scrape.rs): expand templates, then unchanged fetch/selectors. Document one clear rule for when client-supplied `url` is ignored vs overrides DB template (pick one behavior; test).

## Provider backlog + S3

- Table `data_source_provider_requests` + [`src/data/provider_requests.rs`](C:/Users/floris/Documents/GitHub/dexter-sentinel/src/data/provider_requests.rs) + routes (503 via guard).
- Screenshot: URL validation, chromiumoxide, single S3 client path, optional link to provider row.

## API for overrides (thin)

- GET list (debug), POST reload; optional **one** `PUT /api/scraper-overrides/{name}` accepting full `ScraperConfig` JSON (avoids field-by-field drift). Mutations may alternatively stay SQL-only if you prefer ops workflow.

## Docs

- README: override priority, reload curl, AWS/S3/Chrome env, prefetch flag.

Full narrative + diagram lives in [`.cursor/plans/provider_requests_and_s3_screenshots_4797409b.plan.md`](C:/Users/floris/Documents/GitHub/dexter-sentinel/.cursor/plans/provider_requests_and_s3_screenshots_4797409b.plan.md).
