Skip to main content

Module batch

Module batch 

Source
Expand description

Streaming multipart/mixed parser for POST /scrape/batch.

The API emits one part per scrape as each completes. This module reads the response body as a stream of Bytes chunks and yields (headers, body) per part as they arrive.

Zero new dependencies — only reqwest, bytes, and futures-util that are already in Cargo.toml.

Structs§

BatchOptions
Options for Client::scrape_batch_with_options.
BatchPart
One multipart part: header map (lowercased keys) plus body bytes.
BatchPartStream
Stream adapter: wraps a reqwest bytes stream and yields one BatchPart per multipart section as the body arrives.
BatchProxifiedResponse
A proxified batch part surfaced as a native Response-like value. The part body is the raw upstream response (HTML, JSON, binary, etc.) — not a JSON envelope. reqwest::Response is tied to a live connection so we cannot re-synthesize one from bytes; this struct carries the same fields a caller needs.

Enums§

BatchFormat
Wire format for the per-part body. JSON is the default; Msgpack matches the Scrapfly API’s msgpack negotiation.
BatchOutcome
Per-part outcome yielded by Client::scrape_batch.

Functions§

build_proxified_response
Synthesize a BatchProxifiedResponse from a proxified batch part. Restores the upstream HTTP status from X-Scrapfly-Scrape-Status, merges upstream headers (after stripping the X-Scrapfly-Upstream- prefix) with Scrapfly metadata headers, and exposes the raw body.
decode_part_body
Decode a part body according to its Content-Type. Supports application/json (default) and application/msgpack.
parts_from_response
Convenience: take a reqwest Response whose Content-Type is multipart/mixed and return a typed Stream<Item=BatchPart>.