# hf-fetch-model v0.8.0 Roadmap — V2

**Date:** March 21, 2026
**Status:** Proposed
**Supersedes:** [v0.8.0-roadmap.md](v0.8.0-roadmap.md) (V1, March 17, 2026)
**Context:** Dogfooding hf-fetch-model while building anamnesis (tensor format transformation crate) revealed that the tool's biggest gap is *discoverability* — there is no way to see what a remote repo contains without attempting a download. V2 narrows v0.8.0 to two deliverables that close this gap: a `list-files` subcommand and a `--dry-run` download plan. The remaining V1 features are redistributed to future milestones.

---

## Goals

1. **`list-files` subcommand** — inspect remote repo contents (filenames, sizes, checksums) without downloading
2. **`--dry-run` / download plan** — preview what would be downloaded, compare against local cache, and compute an optimal download configuration

---

## Dogfooding Observations Addressed

| # | Observation | Action in v0.8.0 |
|---|------------|-------------------|
| 1 | Missing `list-files` subcommand — no way to see remote repo contents without downloading | New `hf-fm list-files` subcommand (see below) |
| 2 | Main command help is unclear — doesn't explain what gets downloaded | Add a one-liner to `hf-fm` help: *"Downloads all repository files by default; use `--preset safetensors` to download only safetensors, config, and tokenizer files"* |
| 3 | `search --exact` works well | No action needed (positive signal) |
| 4 | `download-file` help is clear | No action needed (positive signal) |

---

## Feature: `list-files` Subcommand

### Why

There is no way to see what files a remote HuggingFace repo contains without attempting a download. `hf-fm download-file` requires knowing the exact filename upfront. A lightweight `list-files` subcommand fills this gap. It reuses the same metadata fetch path as `--dry-run` (`list_repo_files_with_metadata` in `repo.rs`), making them natural companions in one release.

### CLI

```
hf-fm list-files <REPO_ID> [--revision <REV>] [--token <TOKEN>]
    [--filter <GLOB>] [--exclude <GLOB>] [--preset <PRESET>]
    [--no-checksum] [--show-cached]
```

| Flag | Effect |
|------|--------|
| `--filter`, `--exclude`, `--preset` | Same glob-based filtering as the download command. Shows only files that match. |
| `--no-checksum` | Suppresses the SHA256 column. |
| `--show-cached` | Adds a column showing whether each file exists in the local HF cache. |
| `--revision` | Target branch, tag, or commit (default: `main`). |
| `--token` | HuggingFace API token (or `HF_TOKEN` env var). |

### Default output

Filename + human-readable size + LFS SHA256 (truncated to 12 hex characters for readability).

```
$ hf-fm list-files google/gemma-2-2b-it --preset safetensors
  File                                   Size      SHA256
  model-00001-of-00002.safetensors       2.5 GB    a1b2c3d4e5f6
  model-00002-of-00002.safetensors       2.6 GB    f6e5d4c3b2a1
  config.json                            1.2 KB    —
  tokenizer.json                         4.1 MB    —
  ──────────────────────────────────────────────────────────
  4 files, 5.1 GB total
```

Files without LFS metadata (small files stored directly in Git) show `—` in the SHA256 column.

### With `--show-cached`

```
$ hf-fm list-files google/gemma-2-2b-it --preset safetensors --show-cached
  File                                   Size      SHA256        Cached
  model-00001-of-00002.safetensors       2.5 GB    a1b2c3d4e5f6  ✓
  model-00002-of-00002.safetensors       2.6 GB    f6e5d4c3b2a1  ✗
  config.json                            1.2 KB    —             ✓
  tokenizer.json                         4.1 MB    —             ✗
  ──────────────────────────────────────────────────────────────────────
  4 files, 5.1 GB total (2 cached)
```

### Library API

The existing `RepoFile` struct in `repo.rs` already carries the required fields:

```rust
pub struct RepoFile {
    pub filename: String,
    pub size: Option<u64>,
    pub sha256: Option<String>,
}
```

The public entry point is the existing `list_repo_files_with_metadata()` function. The CLI subcommand adds filtering (reusing `config::file_matches`), formatting, and the optional cache-status check (reusing `cache.rs` resolution).

No new library types are needed — `list-files` is a CLI feature that composes existing library primitives.

---

## Feature: Dry-Run / Download Plan

### Why

Before committing to a multi-GB download, users want to know: how many files, total size, what's already cached. More importantly, the plan has all the information needed to compute an *optimal* download configuration — turning the dry-run from a passive preview into an active optimization step. Downloading models is energy-intensive; anything that avoids wasted bandwidth and redundant connections matters.

### CLI

```
hf-fm <REPO_ID> --dry-run [--preset <PRESET>] [--filter <GLOB>] [--exclude <GLOB>]
```

### Behavior

Fetches the file listing and metadata (sizes, checksums) but downloads nothing. Compares against local cache to classify each file. Then computes an optimal `FetchConfig` based on the file size distribution of the *remaining* (uncached) files.

```
$ hf-fm google/gemma-2-2b-it --preset safetensors --dry-run
  Repo:     google/gemma-2-2b-it
  Revision: main (abc1234)
  Filter:   *.safetensors

  File                                   Size      Status
  model-00001-of-00002.safetensors       2.5 GB    cached ✓
  model-00002-of-00002.safetensors       2.6 GB    to download
  ──────────────────────────────────────────────────────────
  Total: 5.1 GB (2 files, 1 cached, 1 to download)
  Download: 2.6 GB

  Recommended config:
    concurrency:        1  (single large file remaining)
    connections/file:   8  (file exceeds chunk threshold)
    chunk threshold:  100 MiB
```

### Plan-to-config optimization strategy

`DownloadPlan::recommended_config()` analyzes the uncached files and tunes:

| Signal | Optimization | Rationale |
|--------|-------------|-----------|
| Few large files (e.g., 1-2 files > 1 GB) | Lower `concurrency`, maximize `connections_per_file` | Saturate bandwidth on fewer streams; avoid connection contention |
| Many small files (e.g., 20 files < 10 MB) | Higher `concurrency`, single connection per file | Chunked overhead exceeds benefit for small files; parallelize at file level instead |
| Mixed sizes | Moderate `concurrency`, chunked only for files above threshold | Balance both strategies |
| All files cached | No download needed | `recommended_config()` returns a config with an empty skip-list; `download_with_plan` is a no-op |
| Single very large file (> 5 GB) | Increase `connections_per_file` to 16 | More parallelism for very large transfers |

The user can always override: `plan.recommended_config()` returns a `FetchConfig` that can be further customized via the builder before passing to `download_with_plan`.

### Two-step API (plan then execute)

```rust
// Step 1: compute the plan (metadata only, no downloads)
let plan = hf_fetch_model::download_plan(repo_id.clone(), &config).await?;

// User inspects: plan.download_bytes, plan.files, etc.

// Step 2a: execute with recommended config
let optimized = plan.recommended_config();
let outcome = hf_fetch_model::download_with_plan(&plan, &optimized).await?;

// Step 2b: or override specific settings
let custom = plan.recommended_config_builder()
    .concurrency(2)       // user override
    .on_progress(|e| { /* ... */ })
    .build()?;
let outcome = hf_fetch_model::download_with_plan(&plan, &custom).await?;
```

### Implicit plan (zero-friction default)

When the user calls `download_with_config` directly (no explicit dry-run), the download pipeline internally computes a plan and applies the recommended config, merged with the user's explicit settings. User-specified values always win; the plan only fills in unset fields. This means **every download benefits from plan-based optimization**, even without `--dry-run`.

### Types

```rust
pub struct DownloadPlan {
    /// Repository identifier.
    pub repo_id: String,
    /// Resolved revision (commit hash).
    pub revision: String,
    /// Per-file plan entries.
    pub files: Vec<FilePlan>,
    /// Total bytes across all files.
    pub total_bytes: u64,
    /// Bytes already cached locally.
    pub cached_bytes: u64,
    /// Bytes that need downloading.
    pub download_bytes: u64,
}

pub struct FilePlan {
    /// Filename within the repository.
    pub filename: String,
    /// File size in bytes.
    pub size: u64,
    /// Whether the file is already cached locally.
    pub cached: bool,
}

impl DownloadPlan {
    /// Compute an optimized `FetchConfig` based on the size distribution
    /// of uncached files. Returns a ready-to-use config.
    pub fn recommended_config(&self) -> FetchConfig;

    /// Like `recommended_config`, but returns a builder so the caller
    /// can override specific fields before finalizing.
    pub fn recommended_config_builder(&self) -> FetchConfigBuilder;

    /// Number of files that need downloading.
    pub fn files_to_download(&self) -> usize;

    /// Whether all files are already cached (download would be a no-op).
    pub fn fully_cached(&self) -> bool;
}
```

---

## Help Text Clarification

Update the main command's clap `about` / `long_about` to clarify download behavior for first-time users:

**Before:** (current — no explanation of what gets downloaded)

**After:**
> Downloads all files from a HuggingFace model repository. Use `--preset safetensors` to download only safetensors weights, config, and tokenizer files.

This addresses dogfooding observation #2 with a one-liner fix.

---

## Shared Metadata Path

Both `list-files` and `--dry-run` rely on the same metadata fetch:

```
GET https://huggingface.co/api/models/{repo_id}?blobs=true
```

This is already implemented as `list_repo_files_with_metadata()` in `repo.rs`, which returns `Vec<RepoFile>` with filename, size, and optional SHA256. The two features diverge only in presentation:

| Feature | Input | Processing | Output |
|---------|-------|-----------|--------|
| `list-files` | `Vec<RepoFile>` | Apply glob filters, resolve cache status | Formatted table |
| `--dry-run` | `Vec<RepoFile>` | Apply glob filters, resolve cache status, compute `DownloadPlan` | Plan table + recommended config |

This shared path means `list-files` is implemented first and `--dry-run` builds on top of it.

---

## Deferred Features

The following V1 features are **not** in v0.8.0:

| V1 Feature | Destination | Rationale |
|-----------|-------------|-----------|
| Feature 1: Safetensors remote header inspection | Deferred — revisit during anamnesis development | New capability independent of download pipeline; will be scoped when anamnesis needs drive clearer requirements |
| Feature 2: Cache management (`du`, `verify`, `clean`) | **v0.9.0** | Natural next step after v0.8.0 adds cache-awareness (`--show-cached`, `--dry-run` cache status) |
| Feature 4: NPZ/NPY download integration | Absorbed by **anamnesis** crate | Pure parsing lives in anamnesis; hf-fetch-model's download integration is a thin wrapper that can be added when anamnesis stabilizes |

---

## Implementation Order

| Order | Deliverable | Rationale |
|-------|------------|-----------|
| 1 | `list-files` subcommand | Simplest feature; exercises and validates the metadata fetch path that `--dry-run` depends on. Delivers immediate value. |
| 2 | `--dry-run` + `DownloadPlan` + plan-to-config | Builds on the metadata path proven by `list-files`. Introduces the `DownloadPlan` type and optimization logic. |
| 3 | Implicit plan retrofit | Wire `DownloadPlan::recommended_config()` into `download_with_config` so every download benefits from plan-based optimization. |
| 4 | Help text clarification | One-liner fix; can land at any point but grouped last to keep focus on the two main features. |

---

## Version Milestone

| Version | Content |
|---------|---------|
| v0.8.0 | `list-files` subcommand, `--dry-run` flag with `DownloadPlan` + plan-to-config pipeline, implicit plan retrofit into `download_with_config`, help text clarification |
| | **Release:** set `version = "0.8.0"` in Cargo.toml, commit, `git tag v0.8.0`, `git push origin v0.8.0` |
