# Changelog

All notable changes to this project will be documented in this file.

## [Unreleased]

## v0.23.0 -- 2026-05-12

### Added
- `OneIo::to_lines_lossy` — lossy line iterator from any `Box<dyn Read + Send>`
- `read_lines_lossy` / `OneIo::read_lines_lossy` — line iteration with invalid UTF-8 replaced by `U+FFFD`
- `read_to_string_lossy` / `OneIo::read_to_string_lossy` / `read_to_string_lossy_async` — full-file read with lossy UTF-8
- `read_to_bytes` / `OneIo::read_to_bytes` / `read_to_bytes_async` — byte-perfect read without UTF-8 validation
- `--strict-utf8` CLI flag for strict UTF-8 validation in the CLI

### Changed
- CLI now defaults to lossy UTF-8 reading; no longer exits on invalid byte sequences
- Documentation examples updated to use lossy APIs

### Deprecated
- `read_lines` — use `read_lines_lossy` for lossy text, `read_to_bytes` for byte-perfect whole-file reads, or `get_reader` + `BufReader` for streaming byte processing
- `read_to_string` — use `read_to_string_lossy` or `read_to_bytes`
- `read_to_string_async` — use `read_to_string_lossy_async` or `read_to_bytes_async`

## v0.22.0 -- 2026-05-01

### Breaking changes
- Migrated S3 implementation from `rust-s3` to `rusty-s3` (sans-IO signing + reqwest HTTP)
  - `s3_bucket()` now returns `S3Bucket` instead of `rust_s3::Bucket`
  - `s3_stats()` now returns `S3ObjectMetadata` instead of `rust_s3::serde_types::HeadObjectResult`
  - Removed re-export of `rust-s3` types; all S3 types are now owned by oneio
- `s3_list()` keeps returning `Vec<String>` for backward compatibility

### Changed
- S3 operations now use the same reqwest HTTP client as HTTP operations (no duplicate HTTP stack)
- Single PUT uploads for files smaller than the multipart threshold (default: 5MB)
- Multipart uploads use 8MB chunks with auto-calculated part sizing for very large files
- `S3Config` is now a public type with endpoint/region fields
- Added `ONEIO_S3_MULTIPART_THRESHOLD` environment variable to configure when to use multipart upload (default: 5MB, the S3 minimum part size)

### Bug fixes
- Fixed `s3_copy()` to use AWS Signature V4 Authorization header for S3-compatible services like R2
- Fixed `digest` command to work with remote URLs (HTTP, HTTPS, FTP, S3)
- Fixed `s3_list()` to decode URL-encoded keys returned by `ListObjectsV2`
- Fixed `oneio s3 list` CLI flag parsing by removing the `-d` short-flag collision

## v0.21.0 -- 2026-03-27

### Breaking changes
- `OneIoError` is now `#[non_exhaustive]`; `match` expressions without a wildcard `_` arm will fail to compile
- `OneIoBuilder::header()` now accepts typed `HeaderName`/`HeaderValue` (infallible) instead of `(K, V) -> Result<Self>`
- `OneIoBuilder::user_agent()` now accepts a typed `HeaderValue` (infallible) instead of `V -> Result<Self>`
- `oneio::download()` no longer accepts an `Option<reqwest::blocking::Client>` parameter
- `oneio::remote` module is now `pub(crate)`; `create_client_with_headers` is deprecated (use `OneIo::builder().header_str()`)
- `ProgressReader` and `ProgressCallback` are no longer part of the public API

### Changed
- Flattened module layout: `src/oneio/` sub-directory removed; all modules are now at `src/` level
- `OneIo` and `OneIoBuilder` are now the primary API surface; free-standing functions delegate to a shared default client
- Compression detection strips URL query parameters and fragments before reading the file extension
- `download_with_retry()` uses exponential backoff between retry attempts (100ms × 2^attempt, capped at 6400ms)
- Stream cache writes to disk via `std::io::copy` instead of buffering the full payload in memory
- `download_async()` now preserves raw bytes, matching `download()`
- Default blocking HTTP clients are reused across reads and content-length probes
- Stateless read and download helpers now delegate to a reusable `OneIo` client internally
- S3 status failures now use structured errors instead of string parsing
- S3 readers now stream data through a bounded channel instead of materializing the full object in memory

### Added
- `OneIoBuilder::header_str(name, value)` — string convenience for adding headers (panics on invalid input, matching reqwest convention)
- `OneIoBuilder::configure_http(f)` — escape hatch for setting any `reqwest::blocking::ClientBuilder` option
- `OneIoBuilder::timeout()`, `connect_timeout()` — request and connect timeouts
- `OneIoBuilder::proxy()`, `no_proxy()` — proxy configuration
- `OneIoBuilder::redirect()` — redirect policy
- `OneIoBuilder::add_root_certificate_pem()`, `add_root_certificate_der()` — load CA certs from raw bytes
- `OneIo::get_reader_with_type(path, compression)` — explicit compression override, useful for URLs with query parameters
- `OneIo::from_client(client)` — construct a `OneIo` from an existing `reqwest::blocking::Client`
- `OneIoError::NetworkWithContext` — network errors now carry the URL that failed
- `OneIoError::InvalidHeader`, `OneIoError::InvalidCertificate` — specific error variants for header and certificate construction failures
- `ONEIO_CA_BUNDLE` environment variable — path to a PEM file added to the HTTP trust store on startup
- `get_cache_reader()` free-standing shortcut kept at crate root for convenience
- Added `bzip2_decompress` benchmark coverage
- Added a benchmark helper script for comparing gzip backend feature flags and bz2 decompression
- Added reusable `OneIo` and `OneIoBuilder` APIs for sharing headers and TLS certificate configuration across requests

### CLI
- Added `-H`/`--header` flag for custom HTTP headers (`"Name: Value"` or `"Name:Value"`), can be repeated
- Added `--compression` flag to override compression detection (gz, bz2, lz4, xz, zst); no effect with `--download`
- Added progress bar for `--download` and `s3 download`, shown when stderr is a terminal; uses spinner when file size is unknown
- Added `s3 download <BUCKET> <S3_PATH> [--outfile]` subcommand
- Fixed S3 upload syntax: `oneio s3 upload <LOCAL_FILE> <BUCKET> <S3_PATH>` (local file is now the first positional arg under the subcommand)
- Terminal detection uses `std::io::IsTerminal` from the standard library; no extra dependency needed
- Added `indicatif` to the `cli` feature for progress bars

### Bug fixes
- LZ4 compressed writes were silently truncated: `lz4::Encoder` has no `Drop` impl and requires an explicit `finish()` call to write the end-of-stream marker. Fixed with a `Lz4Writer` wrapper that calls `finish()` on drop.

### Documentation
- `lib.rs` docstring documents the `native-tls` feature as the fix for Cloudflare WARP and corporate proxy environments
- `ONEIO_ACCEPT_INVALID_CERTS` and `ONEIO_CA_BUNDLE` environment variables documented at crate root

## v0.20.1 -- 2025-12-18

### Changed
- `rustls` feature now enables both `rustls-tls-native-roots` and `rustls-tls-webpki-roots` for reqwest
  - Provides support for system certificates (including those installed by Cloudflare WARP or corporate VPNs)
  - Falls back to bundled Mozilla certificates in minimal environments
  - Both certificate stores are merged, providing maximum compatibility

## v0.20.0 -- 2025-10-29

### Added
- New `crypto` module with `ensure_default_provider()` helper function to initialize rustls crypto providers
  - Automatically tries AWS-LC first, then falls back to ring
  - Called automatically from all HTTPS/S3/FTP code paths
  - Can be called explicitly at startup for better control over initialization
  - Fully thread-safe and idempotent
  - Prevents "Could not automatically determine the process-level CryptoProvider" panic
- GitHub Copilot custom instructions file (`.github/copilot-instructions.md`) documenting development practices

### Changed
- All HTTPS, S3, and FTP operations now automatically initialize the rustls crypto provider on first use
- Added comprehensive documentation and examples for crypto provider initialization
- `rustls` feature now explicitly includes `dep:rustls_sys` dependency
- Simplified redundant feature flag checks (since `https` → `rustls`, only check `rustls`)
- Split CI workflow: formatting check now runs as separate independent job for faster feedback
- Updated `remote_file_exists()` to use `ensure_default_provider()` helper

## v0.19.2 -- 2025-10-17

### Changed

- Updated `flate2` dependency minimum version to `1.1.0` to include more recent fixes for `zlib-rs` backend.
- Add more backends for `flate2`:
  - `gz-zlib-ng` feature for `flate2/zlib-rs-ng`
  - `gz-zlib-cloudflare` feature for `flate2/cloudflare_zlib`

## v0.19.1 -- 2025-10-15

### Changed
- Gzip backend selection via feature flags:
  - Default feature `gz` switched to use flate2 with the `gz-zlib-rs` backend for improved performance.
  - New selectors and aliases:
    - `gz-zlib-rs` — enables `flate2/zlib-rs` (Rust, fast)
    - `gz-miniz` — enables `flate2/miniz_oxide` (pure Rust, most portable)
  - Disabled `flate2` default-features to allow explicit backend choice.

### Added
- Criterion benchmark `benches/gzip_decompress.rs` to measure gzip decompression throughput across backends.

### Usage
- Default (zlib-rs):
  - cargo build
  - cargo bench --bench gzip_decompress --features gz
- zlib-rs:
  - cargo build --no-default-features --features gz-zlib-rs
  - cargo bench --bench gzip_decompress --no-default-features --features gz-zlib-rs
- miniz_oxide (explicit):
  - cargo build --no-default-features --features gz-miniz
  - cargo bench --bench gzip_decompress --no-default-features --features gz-miniz

## v0.19.0 -- 2025-08-31

### Breaking Changes

#### Feature Flag Simplification

- Removed complex nested feature structure (`lib-core`, `remote`, `compressions`)
- Introduced flat structure: `gz`, `bz`, `lz`, `xz`, `zstd`, `http`, `ftp`, `s3`, `async`
- Changed default features from `["lib-core", "rustls"]` to `["gz", "bz", "http"]`

**Migration guide:**

```
# Before (v0.18.x)
oneio = { version = "0.18", features = ["lib-core", "rustls"] }

# After (v0.19.0)  
oneio = { version = "0.19", features = ["gz", "bz", "http"] }
```

#### Error System Consolidation

- Simplified from 10+ error variants to 3 essential categories:
    - `Io(std::io::Error)` - File system errors
    - `Network(Box<dyn Error>)` - Network/remote errors
    - `NotSupported(String)` - Feature is not compiled or unsupported operation

#### Removed Components

- Removed `build.rs` - No longer needed with simplified feature structure
- Removed `OneIOCompression` trait - Replaced with direct function calls

### New Features

#### Progress Tracking

- Added `get_reader_with_progress()` for tracking download/read progress
- Works with both known and unknown file sizes
- Tracks raw bytes read before decompression
- Callback-based API: `|bytes_read, total_bytes| { ... }`

#### Async Support (Feature: `async`)

- Added `get_reader_async()`, `read_to_string_async()`, `download_async()`
- True async support for HTTP and local files
- Compression support for gzip, bzip2, and zstd via async-compression
- LZ4 and XZ return `NotSupported` errors (no native async support)
- FTP/S3 protocols return clear "not supported" errors

#### CLI Enhancements

- Added LZ/XZ compression support to CLI tool
- Fixed typos in CLI option descriptions

### Improvements

#### Code Quality

- Replaced unsafe `unwrap()` calls in path parsing with safe alternatives
- Improved FTP login and file operation error handling
- Simplified HTTP stream error mapping
- Enhanced LZ4 error handling with better context
- Removed `eprintln` from library code in progress tracking

#### S3 Improvements

- Replaced fragile string matching with HTTP status code parsing in `s3_exists()`
- Fixed S3 upload hanging on non-existent files (issue #48)
- Made S3 documentation tests conditional on s3 feature

#### Testing and Examples

- Tests now work with any feature combination, including no features
- Updated progress tracking example to use indicatif
- Replaced unreliable `httpbin.org` endpoint with stable `spaces.bgpkit.org` endpoint
- Fixed doctest compilation with appropriate `ignore` flags

#### Documentation

- Updated lib.rs documentation for v0.19 changes
- Clarified `http` vs `https` feature distinctions:
  - `http` - HTTP-only support (no TLS)
  - `https` - HTTP/HTTPS with rustls (equivalent to `http` + `rustls`)
  - Custom TLS: Use `http` + `rustls` or `http` + `native-tls`
- Regenerated README from lib.rs documentation
- Added clear migration guide and feature explanations

### Dependencies

#### Updated

- Upgraded bzip2 to 0.6.0
- Added indicatif to dev-dependencies

#### Added (for async feature)

- tokio
- async-compression
- futures

### Code Simplification

- Removed trait-based compression system in favor of direct function calls
- Eliminated nested feature dependencies with a flat structure
- Simplified async implementation without spawn_blocking patterns
- Streamlined error types from 10+ variants to 3 categories

## v0.18.2 -- 2025-06-06

### Hot Fix

* Make `rustls_sys` dependency optional and exclude from `rustls` feature flag

## v0.18.1 -- 2025-05-31

### ✨ Added

- **New build script**: Added `build.rs` to enforce that at least one TLS backend (`rustls` or `native-tls`) is enabled
  if any of the remote features (`http`, `ftp`, or `remote`) are enabled.
- **Module documentation**: Added detailed Rust doc comments to the compression modules (gzip, bzip2, lz4, xz, zstd) and
  `utils.rs` for improved usability and understanding.
- **`get_protocol` function**: Utility for extracting protocol from file paths, now used across remote file access
  functions.

### 🛠️ Changed

- **Feature dependencies**: The `ftp` feature now explicitly depends on the `http` feature in `Cargo.toml`.
- **Error handling**: Updated `OneIoError` enum to more accurately gate error variants with corresponding features (
  `http`, `ftp`).
- **Module structure**:
    - `compressions` is now a public module.
    - Refactored how the crate distinguishes between local and remote file access, using `get_protocol`.
    - `get_reader_raw` and related functions now determine protocol and select the appropriate file reader accordingly.
- **Compression interface**:
    - Added a unified trait `OneIOCompression` and `get_compression_reader`/`get_compression_writer` utilities for
      consistent handling of all supported compression algorithms.
    - Updated file open logic to use these helpers based on file suffix.

### 🧹 Cleaned up and Improved

- Removed legacy or redundant code paths (e.g., `get_reader_raw_remote`, old error gates).
- Moved protocol detection and remote file reading logic into more modular and maintainable forms.
- Several function signatures and internal APIs have been updated for clarity and maintainability.

---

### 📝 Developer Notes

- All compression modules (`gzip`, `bzip2`, `lz4`, `xz`, `zstd`) now include clear documentation and consistent
  interfaces for reading and writing compressed files.
- Users enabling remote protocol features must ensure at least one TLS backend is also enabled.

## v0.18.0 -- 2025-05-30

### Highlights

* split `remote` features into `http` and `ftp`, allowing users who only need HTTP or FTP support to use the
  corresponding feature flag
    * in most cases, users will likely not need to use the `ftp` feature
* add `create_client_with_headers` function to allow creating a `reqwest::blocking::Client` with custom headers
    * this simplifies the process of creating a client with custom headers for HTTP requests
    * this also allows users to create custom clients without importing `reqwest` crate directly
* add `rustls_sys` dependency to support `rustls` as the default TLS backend

### Documentation improvements

* update examples on custom HTTP request function `oneio::get_http_reader` to use `create_client_with_headers`

## v0.17.0 -- 2024-08-04

### Highlights

* add support for `zstd` (zstandard) compression
* allow setting `ONEIO_ACCEPT_INVALID_CERTS=true` to disable SSL checking
* revised custom HTTP request function `oneio::get_http_reader` to allow specifying custom `request::blocking::Client`
  for any request customizations to allow specifying custom `request::blocking::Client` for any request customizations.

### Breaking changes

1. rename `oneio::get_remote_reader` to `oneio::get_http_reader`
2. rename `get_remote_ftp_raw` to `get_ftp_reader_raw`
3. change signatures of `oneio::download`, `oneio::download_with_retry`, `oneio::get_http_reader`'s optional HashMap
   parameter for headers to optional `reqwest::blocking::Client`.

## v0.16.8 -- 2024-05-22

### Highlights

* `s3_url_parse` allow parsing different protocols like `r2://` or `b2://`
    * previously, if the URL did not start with `s3://` it would return an error

## v0.16.7 -- 2024-03-26

### Highlights

* make `compressions` mod always enabled and allow making all compression algorithms optional
    * to enable all compression algorithms,
      use `oneio = { version = "0.16", default-features = false, features = ["compressions"] }`
    * to enable specific compression algorithms,
      use `oneio = { version = "0.16", default-features = false, features = ["gz", "bz", "lz", "xz"] }`

## v0.16.6 -- 2024-03-26

### Highlights

* make `digest` feature optional and disabled by default

## v0.16.5 -- 2024-03-20

### Highlights

* add `CONTENT_LENGTH=0` to headers to address some queries where the server request `Content-Length` field

## v0.16.4 -- 2024-03-20

### Hot fix

* add `http2` and `charset` feature flags to `reqwest`
    * the feature flags for `reqwest` has changed a lot between `0.11` and `0.12` and the `http2` and `charset` features
      are necessary now

## v0.16.3 -- 2024-03-20

* switch `flate2` to `rust-backend` default feature as the `zlib-ng` feature requires `cmake` to build and offers no
  performance improvement over the `rust-backend` feature
* update `reqwest` to version `0.12`

## v0.16.2 -- 2024-02-23

### Highlights

* switching to `flate2` with `zlib-ng` as the compression library for handling `gzip` files
    * `zlib-ng` is a drop-in replacement for `zlib` with better performance

## v0.16.1 -- 2024-02-10

### Highlights

* add `oneio::exists(path: &str)` function to check if a local or remote file exists.
    * currently support local file, http(s) remote and s3 remote files checking

### Example usages

```rust
assert!(!oneio::exists("https://spaces.bgpkit.org/oneio/test_data_NOT_EXIST.json").unwrap());
assert!(oneio::exists("https://spaces.bgpkit.org/oneio/test_data.json").unwrap());
```

## v0.16.0 - 2024-01-29

### Breaking changes

- switch to `rustls` as the default TLS backend
- clean up the feature flags
    - removed `no-cache` and `vendored-openssl` flags
    - removed `openssl` optional dependency
    - add `digest` feature flag to allow calculating SHA256 digest of a file, enabled by default

### Library changes

- add `oneio::download_with_retry` function to allow retrying download
- add `oneio::get_sha256_digest` function to the library to calculate SHA256 digest of a file

### CLI changes

- add `oneio digest FILE` command to calculate file SHA256 digest

## v0.15.10 - 2024-01-26

### Hot fix

- fixed issue where `oneio s3 list BUCKET PREFIX` command not showing files match the prefix unless they are on the same
  directory as the prefix
- fixed issue where running `oneio` without argument causing program to panic

## v0.15.9 - 2024-01-26

### Highlights

- add support for installing via [`cargo binstall`](https://github.com/cargo-bins/cargo-binstall)
- `s3_list` now accepts a new forth parameter, `dirs` boolean flag, to allow returning only the matching directories
- add `oneio s3 list` and `oneio s3 upload` commands to the CLI

### Breaking changes

The signature of `s3_list` function is now changed to:

```rust
pub fn s3_list(
    bucket: &str,
    prefix: &str,
    delimiter: Option<String>,
    dirs: bool,
) -> Result<Vec<String>, OneIoError> {}
```

This includes changes of:

1. `delimiter` changed from `Option<&str>` to `Option<String>`
2. new `dirs` boolean flag to allow returning only matching directories in the specified prefix
    - the `delimiter` is also automatically forced to `Some("/".to_string())` if `dirs` is specified and `delimiter` is
      specified as `None`.

### Misc

- "cargo publish" process is now automated with GitHub actions
- test process now makes sure `s3` modules' doc-tests must compile

## v0.15.8 - 2023-12-18

### Highlights

* Added `vendored-openssl` flag to enable GitHub actions builds for different systems.
* Automatically builds CLI binary for macOS (Universal), and linux (arm and amd64) during GitHub release

## v0.15.8-beta.1 - 2023-12-16

### Highlights

* GitHub actions use vendored openssl instead of system openssl.

## v0.15.7 - 2023-12-16

### Highlights

* Module Refactoring: A dedicated module has been created for remote and utils.
  (ec80e0236170f13e9eec2450eeaa8334e255a1ee)
* Improvements in Caching Control: The HTTP caching is now controllable and disabled for CLI requests.
  (69de57c5f9a7003edecde2fe6641c438abe233a5)
* Improved Error Handling: We have improved error handling in line reads. The application no longer attempts to read
  further if a line read fails, preventing any stalls. (fd1352fa2cb701e3fb336a4b6f99014d76d64788)

## v0.15.6 - 2023-12-16

### Added

- support getting oneio reader directly by supplying an S3 URL: https://github.com/bgpkit/oneio/pull/31
