# scanclient

[![beta](https://img.shields.io/badge/status-beta-yellow)](https://github.com/santhsecurity/scanclient)
[![crates.io](https://img.shields.io/crates/v/scanclient.svg)](https://crates.io/crates/scanclient)
[![docs.rs](https://docs.rs/scanclient/badge.svg)](https://docs.rs/scanclient)

## What it does

`scanclient` is a connection-pooled, retry-aware asynchronous HTTP client tuned
for security scanners. It wraps `reqwest` and adds:

- **Bounded response bodies** – configurable `max_body_size` cap (default 10 MiB)
  enforced on both `Content-Length` and the streaming read.
- **Status-aware retries** – exponential back-off on 429, 408, 500, 502, 503, 504
  with optional `Retry-After` header respect.
- **Redirect SSRF protection** – redirect targets are checked against the bogon
  list before connecting; private/loopback addresses are blocked by default.
- **Rate limiting** – per-client token-bucket gate via `rate_limit_per_sec`.
- **Proxy support** – HTTP/SOCKS5 proxies with optional `Proxy-Authorization`.
- **Custom headers and User-Agent rotation**.
- **TLS options** – strict verification by default; opt-in relaxation for lab
  environments.
- **URL utilities** – `resolve_url`, `clean_url`, `registrable_domain`,
  `dedup_host_key`, `strip_www_host` – canonical implementations shared across
  the Santh scanner fleet.

## Quick start

```rust
use scanclient::config::HttpConfig;

// Build from defaults (all values documented in the table below)
let config = HttpConfig::default();
assert_eq!(config.timeout_secs, 10);
assert!(config.tls_verify);

// Load from TOML
let config = HttpConfig::from_toml(r#"
    timeout_secs = 30
    max_body_size = 10485760
    max_retries = 5
    proxy = "http://127.0.0.1:8080"
    proxy_user = "scanner"
    proxy_password = "secret"
    tls_verify = false
    rate_limit_per_sec = 10

    [custom_headers]
    X-Bug-Bounty = "authorized"
"#).unwrap();
```

Async GET:

```rust,no_run
#[tokio::main]
async fn main() {
    use scanclient::{HttpConfig, client::ScanClient};
    let client = ScanClient::from_config(HttpConfig::default()).unwrap();
    let resp = client.get("https://example.com/").await.unwrap();
    println!("{}", resp.status());
}
```

## When to use / When not

**Use `scanclient` when you need:**

- Scan-safe operational controls: bounded reads, SSRF-safe redirects, retries.
- A consistent config pattern (TOML-loadable) across multiple scanner binaries.
- Rate limiting and concurrency control against a single target host.
- Proxy auth wired up without boilerplate.

**Do not use `scanclient` when:**

- You need a general-purpose HTTP client without scanner-specific policy
  (use `reqwest` directly).
- You are streaming large bodies that should not be fully buffered in memory
  (increase `max_body_size` or stream outside this crate).
- You need full RFC 3986 URL resolution with `..` segment collapsing
  (`urlutil::resolve_url` is string-level; use the `url` crate directly for
  those cases).

## Compared to alternatives

| | scanclient | reqwest (direct) | ureq |
|---|---|---|---|
| Bounded body reads | yes (enforced) | manual | manual |
| SSRF-safe redirect policy | yes | no | no |
| TOML-loadable config | yes | no | no |
| Exponential back-off retries | yes | no | no |
| TLS impersonation | optional feature | no | no |
| Async | yes | yes | no |

`scanclient` is scanning-focused: rate limiting, target management, safe redirect
handling, and finding-friendly response collection.

`StealthClient` (in this same crate, under the `tls-impersonate` feature) is
stealth-focused: wire-identical browser TLS hellos, header rotation, and timing
variation. They compose - a scanner can use `StealthClient` for fingerprint
evasion and wrap results in scan-safe policy via `scanclient`.

## How it fits in Santh

Every Santh scanner binary (`wafrift`, `golemn`, `karyx`, `scald`, `sear`,
`netshift`) depends on this crate for outbound HTTP. It is the single place where:

- The SSRF bogon-block policy is enforced for redirect chains.
- Response body limits are applied before findings are written.
- Retry/back-off policy is tuned for scanner traffic patterns.
- URL canonicalization (`urlutil`) is kept in sync across the fleet.

Configuration is expressed as `HttpConfig` structs (TOML-loadable), so every
scanner inherits consistent defaults and operators can override them from a
single file.

## Configuration reference

Every field has a sensible default. Override what you need:

| Field | Default | What it does |
|-------|---------|-------------|
| `timeout_secs` | 10 | Request timeout |
| `max_body_size` | 10485760 | Maximum buffered response body bytes (10 MiB) |
| `max_retries` | 3 | Retry on transient failure |
| `retry_delay_ms` | 1000 | Base delay between retries (exponential back-off) |
| `max_redirects` | 5 | Follow redirects (bogon-checked) |
| `proxy` | None | HTTP/SOCKS5 proxy URL |
| `proxy_user` | None | Proxy username for `Proxy-Authorization` |
| `proxy_password` | None | Proxy password for `Proxy-Authorization` |
| `user_agent` | Chrome Windows stable stealth profile | Request `User-Agent` |
| `rate_limit_per_sec` | None | Max requests per second |
| `tls_verify` | `true` | Verify TLS certificates |
| `connect_timeout_secs` | 5 | Connection timeout |

## Contributing

Pull requests are welcome. If you find a bug, a better API, or a rough edge,
open an issue or PR. To run the full test suite:

```shell
cargo +nightly test -p scanclient
```

## License

MIT OR Apache-2.0. Copyright 2026 CORUM COLLECTIVE LLC.
