# duckduckgo-search-cli

- Rust CLI for DuckDuckGo search — returns results as JSON, Markdown, or plain text
- Install with: `cargo install duckduckgo-search-cli`
- Current version: v0.7.3
- Repository: https://github.com/daniloaguiarbr/duckduckgo-search-cli


## What's new in v0.7.3

- **GAP-WS-27 fixed (CRITICAL).** The macOS CAPTCHA interstitial that returned HTTP 200 with `quantidade_resultados: 0` while Windows returned full results is closed. Root cause: `rustls` produces a TLS fingerprint recognisable by Cloudflare Bot Management (JA4_o vector). v0.7.3 replaces `reqwest 0.12` + `rustls-tls` with `wreq 6.0.0-rc.29` + BoringSSL (`boring2` 4.15.11) + `wreq-util 3.0.0-rc.12`. BoringSSL produces a JA4_o fingerprint identical to Chrome/Safari. Reproduction: same query, same machine — v0.7.2 returned 0 results in 1695ms; v0.7.3 returns 5 results in 735ms. See `docs/decisions/0001-tls-boring-via-wreq.md`.
- **New `session` feature (cookie persistence + warm-up).** New flags: `--no-warmup` (skip `GET https://duckduckgo.com/` warm-up), `--no-cookie-persistence` (in-memory only), `--cookies-path <PATH>` (override XDG default). Cookie jar is written to `~/.config/duckduckgo-search-cli/cookies.json` (Linux), `%APPDATA%\duckduckgo-search-cli\cookies.json` (Windows), or `~/Library/Application Support/duckduckgo-search-cli/cookies.json` (macOS) with Unix permissions `0o600`.
- **New `probe-deep` feature (CAPTCHA interstitial detection).** New flags: `--probe-deep` (run a real search query and classify the body as `ok` or `captcha` based on Cloudflare and DuckDuckGo markers), `--allow-lite-fallback` (opt-in to automatic `html → lite` fallback when CAPTCHA is detected).
- **Build prerequisites added.** Compiling from source now requires `cmake`, `perl`, `pkg-config`, and `libclang-dev` on Linux. Pre-built binaries from crates.io are unaffected. CI matrix in `.github/workflows/release.yml` installs these packages in the Linux jobs.
- **New modules** — `session_warmup` (XDG path resolution), `wreq_cookie_adapter` (JSON ↔ `wreq::cookie::Jar` bridge), `probe_deep` (interstitial detection).
- **Zero breaking changes to JSON output schema** — all v0.7.2 fields remain present.
- **Test count: 292 lib** (was 279 in v0.7.2) + 18 wiremock + other integrations. 0 clippy warnings, 0 fmt diff, 2 cargo-deny warnings (RUSTSEC-2025-0057 and RUSTSEC-2025-0052, both already in the ignore list).
- **Binary size +20 MB** (BoringSSL is statically linked). Release build time +30s to +90s.


## What's new in v0.7.2

- **Security advisory fix (RUSTSEC-2026-0009)**: `time 0.3.40` denial-of-service via RFC 2822 stack exhaustion was being pulled in transitively via `cookie_store 0.22.0` → `reqwest 0.12.28`. v0.7.2 pins `time = "0.3.47"` as a direct dep to override the transitive constraint.
- **`rand` 0.10 migration**: dev-deps (proptest 1.11+, getrandom 0.4+) unified on rand 0.10. Convenience methods moved from `Rng` to `RngExt`. All internal call sites updated.
- **MSRV bump**: `rust-version` raised from 1.75 to 1.88 (required by `time 0.3.47+` and `rand 0.10`).
- **402 tests passing** (289 lib + 101 integration + 12 doctest), 0 failures.

## What's new in v0.7.1

- **`rand` 0.9 migration** (zero breaking changes). `random_range`, `random_bool`, `random`, and `IndexedRandom::choose`.
- **MSRV bump** to 1.85.
- **reqwest builder cleanup**: `gzip` and `brotli` features removed (now handled via `Accept-Encoding`).
- **392 tests passing**.

## What's new in v0.7.0

- **New subcommand `deep-research`** — query fan-out pipeline for multi-hop LLM research
  - Splits one query into 1..=12 sub-queries via five canonical heuristic templates (aspect, comparison, timeline, opinion, cause) or a manual file
  - Fans out in parallel via the existing `JoinSet` + `Semaphore` executor
  - Aggregates with Reciprocal Rank Fusion (K=60) or canonical-URL deduplication
  - Optional synthesis to Markdown, PlainText, or JSON with numbered references and a token budget
- **Four new public modules** — `deep_research`, `decomposition`, `aggregation`, `synthesis` — composable from downstream crates
- **New dependencies** — `url = "2"` (canonicalisation), `regex = "1"` (composite-query detection), `proptest = "1"` (dev, property tests)
- **Zero breaking changes** — `buscar`, `init-config`, default-config JSON schema, and all exit codes are byte-for-byte identical to v0.6.x
- **392 tests passing** (279 lib + 12 doc + 101 integration), 8 validation gates OK


## What's new in v0.6.5

- **CRITICAL FIX (MP-26)**: Windows build compiles again. `windows-sys 0.59+` changed `HANDLE` from `isize` to `*mut c_void`. Replaced `handle != 0 && handle != usize::MAX` with `!handle.is_null() && handle != INVALID_HANDLE_VALUE`.
- **CI fix (CI-01)**: 6 latent clippy errors in v0.6.4 broke the `validate` job on all 3 SOs. Fixed: `doc_markdown` on 3 strings (`PowerShell`, `rules_rust.md`, `TempDir`), `needless_return`, `missing_debug_implementations` on `ChromeBrowser` and `CircuitBreakerMap`.
- **Preventive FFI lints**: `improper_ctypes = "deny"` and `improper_ctypes_definitions = "deny"` in `Cargo.toml`.
- **WS-11**: 5 property tests (invariants) in `extraction.rs` — no new dependency.
- **WS-12**: Per-host circuit breaker in `content_fetch.rs` — 3 consecutive failures open the circuit for 30s.
- **WS-23**: WireMock test validates the `Retry-After` header on 429 responses.
- **WS-25**: `indicatif = "0.18"` added for the ProgressBar in long crawls.
- **333 tests passing**, 8 validation gates OK (fmt, clippy, test, doc, audit, deny, publish dry-run, package list).


## What's new in v0.6.4

- Adaptive anti-bot identity pool (12 identities, 4 browser families × 3 platforms)
- 5-level cascade rotation when DDG blocks (HTTP 202/403/429)
- New JSON fields: `metadados.identidade_usada` and `metadados.nivel_cascata` (additive, non-breaking)
- New CLI flags: `--probe` (pre-flight health check) and `--identity-profile` (pin profile)
- `--seed` flag now also controls identity pool rotation (previously UA only)


## Canonical Documentation

- `README.md` — overview, flags, exit codes, bilingual examples
- `docs/HOW_TO_USE.md` — pedagogical step-by-step guide
- `docs/COOKBOOK.md` — 23 ready-to-use recipes for search, ETL, monitoring, and deep research
- `docs/INTEGRATIONS.md` — integrations with 16+ AI agents (Claude, GPT, Gemini, Cursor…)
- `docs/AGENTS-GUIDE.md` — stdin/stdout contract and full JSON schema reference
- `docs/CROSS_PLATFORM.md` — setup on Linux, macOS, Windows, and Docker
- `docs/AGENTS.md` — quick integration guide for AI agents
- `docs/AGENT_RULES.md` — 30+ MUST/NEVER rules for production use by agents
- `docs/MIGRATION.md` — migration guide between versions
- `docs/TESTING.md` — test execution and categorization
- `skill/duckduckgo-search-cli-en/SKILL.md` — Claude/OpenCode skill for agents
- `skill/duckduckgo-search-cli-pt/SKILL.md` — Portuguese skill for agents
- `CHANGELOG.md` — version history (EN)
- `CHANGELOG.pt-BR.md` — version history (PT)
- `INTEGRATIONS.md` — pointer to full integration catalog
- `CONTRIBUTING.md` — contributor onboarding
- `CODE_OF_CONDUCT.md` — community standards
- `SECURITY.md` — vulnerability disclosure policy


## Quick Start

### Basic search (plain text on TTY)

```bash
timeout 30 duckduckgo-search-cli "rust async runtime"
```

### Machine-readable JSON (piped)

```bash
timeout 30 duckduckgo-search-cli -q -f json --num 10 "rust async runtime" \
  | jaq -r '.resultados[].url'
```

### Pre-flight health check (v0.6.4+, preserved in v0.6.5)

```bash
timeout 15 duckduckgo-search-cli --probe
# {"endpoint":"html","has_set_cookie":false,"latency_ms":109,"status":200,"type":"probe","url":"https://html.duckduckgo.com/html/"}
```

### Batch mode (multi-query)

```bash
printf 'query one\nquery two\nquery three\n' > /tmp/q.txt
timeout 300 duckduckgo-search-cli \
  --queries-file /tmp/q.txt \
  -q -f json --parallel 3 --per-host-limit 1 --retries 3 \
  --global-timeout 280 > /tmp/multi.json
```

### Content extraction (long crawls, WS-25 ProgressBar)

```bash
timeout 120 duckduckgo-search-cli -q -f json --num 5 \
  --fetch-content --max-content-length 5000 \
  "rust embedded systems" | jaq '.resultados[].url'
```


## Canonical Invocations

```bash
# Single query
duckduckgo-search-cli -q -f json "query" | jaq '.resultados | length'

# Multi-query (use --queries-file, NEVER shell loops)
duckduckgo-search-cli -q -f json "one" "two" | jaq '.buscas[0].resultados | length'

# Filter by URL
duckduckgo-search-cli -q -f json "rust" | jaq -r '.resultados[].url'

# Detect identity used (v0.6.4+)
duckduckgo-search-cli -q -f json "rust" | jaq -r '.metadados.identidade_usada // "n/a"'

# Detect cascade level (v0.6.4+)
duckduckgo-search-cli -q -f json "rust" | jaq '.metadados.nivel_cascata // 0'

# Deep research (v0.7.0+): default heuristic, RRF, no synthesis
timeout 60 duckduckgo-search-cli -q -f json deep-research "rust http client 2026" \
  | jaq '.resultados[] | {titulo, url, score}'

# Deep research with synthesis (Markdown report)
timeout 120 duckduckgo-search-cli -q -f json deep-research "tokio vs async-std 2026" \
  --synthesize --budget-tokens 1500 --synth-format markdown \
  --fetch-content --max-content-length 6000 | jaq '.sintese'

# Deep research with manual sub-queries from a file
cat > /tmp/qs.txt <<EOF
# Visão geral
what is tokio runtime 2026
# Comparação
tokio vs async-std vs smol
EOF
timeout 60 duckduckgo-search-cli -q -f json deep-research "tokio runtime 2026" \
  --sub-queries-file /tmp/qs.txt --aggregate dedupe-by-url \
  | jaq '.metadados.sub_queries | length'
```


## Exit Codes

| Code | Meaning           | Agent action                              |
|------|-------------------|-------------------------------------------|
| `0`  | Success           | Parse `.resultados`                       |
| `1`  | Runtime error     | Read stderr; retry once with `-v`         |
| `2`  | Config error      | Re-run `init-config --force`              |
| `3`  | Anti-bot block    | Back off 300+ s; switch `--endpoint lite` |
| `4`  | Global timeout    | Raise `--global-timeout`; reduce `--parallel` |
| `5`  | Zero results      | Refine query or change `--lang`           |


## Anti-Patterns to Avoid

- NEVER parse `text` or `markdown` output with machines (use `json` + `jaq`)
- NEVER omit `timeout` (pipelines hang indefinitely)
- NEVER pass proxy credentials in argv (use `HTTPS_PROXY` env var)
- NEVER raise `--parallel` above 5 (triggers anti-bot)
- NEVER use `--stream` (placeholder, not implemented)
- NEVER inject custom `Sec-Fetch-*` or `Accept-Language` headers


## License

Dual-licensed under MIT OR Apache-2.0. See `LICENSE` for details.
