# thesa

`thesa` is a Rust CLI to archive GitHub repositories, ML models, and websites.
Website archiving is powered by `siteforge`.

## Install

`cargo install --path .`

From crates.io after publishing:

`cargo install thesa`

Then run:

`thesa`

Or force the interactive TUI explicitly:

`thesa --tui`

Short form:

`thesa -T`

## Usage

`thesa [TARGET] [options]`

New command form:

- `thesa archive site <URL>`
- `thesa archive url <URL>`
- `thesa verify <ARCHIVE_DIR>`

Run with no arguments for interactive mode.

Use `--tui` or `-T` to force interactive mode. If `TARGET` is also passed, the
TUI flag takes precedence and opens the TUI prompt instead.

The TUI landing screen offers three archive routes: GitHub repositories, ML
models, and Siteforge website archives. Selecting Siteforge prompts for a URL
and then shows a preflight options screen for full-site vs single-page archives,
max depth, max pages, and JavaScript rendering. TUI Siteforge runs archive into
`./archives/sites` by default.

`TARGET` can be one of: `owner`, `owner/repo`, `https://github.com/owner`,
`https://github.com/owner/repo`.

For model mode, `TARGET` can also be presets:

- `top` (most downloaded)
- `latest` (most recently updated)

For model mode, set `--mode models` and choose one provider:

- `--provider hf` (Hugging Face, default)
- `--provider ollama` (local Ollama instance)
- `--provider civitai` (CivitAI)

In interactive/TUI model mode, provider selection appears before the model target
prompt.

Common options:

- `--depth <DEPTH>`: clone with commit depth
- `-c, --concurrency <N>`: number of repos cloned in parallel (default `4`)
- `--filter <PATTERN>`: filter repository names by substring
- `--skip-existing`: skip repos that are already present
- `--token <TOKEN>`: GitHub API token (or use `GITHUB_TOKEN` env)
- `--hf-token <TOKEN>`: Hugging Face API token (or use `HF_TOKEN` env)
- `--civitai-token <TOKEN>`: CivitAI API token (or use `CIVITAI_TOKEN` env)
- `--hf-mirror`: use `hf-mirror.com` for Hugging Face API calls
- `--provider <PROVIDER>`: model provider for model mode (`hf`, `ollama`, `civitai`)
- `-T, --tui`: force the interactive Scrin interface

Website archive options:

- `--full-site`: follow in-scope site links instead of archiving one page
- `--allow-cross-domain`: permit cross-domain links
- `--max-depth <N>`: limit crawl depth
- `--max-pages <N>`: limit page count
- `--include <GLOB>`: include URL glob pattern, repeatable
- `--exclude <GLOB>`: exclude URL glob pattern, repeatable
- `--render-js`: use Siteforge browser rendering for JavaScript pages

Repo picker controls (TTY only):

- `Space`: check or uncheck a repo
- Mouse: hover a visible row to focus it, left-click to check or uncheck it
- `a`: check all visible repos
- `c`: clear filter and checks
- `b`: open batch-size overlay (`1`, `2`, `3`, `6`, `8`)
- `PageUp`/`PageDown`: move cursor by one viewport page
- `Home`/`End`: jump to first/last visible result
- `Enter`: clone checked repos only, or visible repos when nothing is checked

Clone dashboard controls (TTY only):

- `t`: toggle repository detail lines in the live batch display
- `q`: request stop after current batch completes
- `m`: motion effect
- `b`: Burn effect
- `w`: Wipe effect

TTY clone runs use a Scrin dashboard. Aisling renders the top Matrix deck,
effect pane, and TQDM-style loader rows for concurrent repository pulls. The TUI
is built against newer Scrin interaction support and remains keyboard-first for
SSH reliability.

Mode, provider, repository, and model picker screens also accept mouse hover and
left-click actions when the terminal supports mouse reporting.

Examples:

- Start TUI and pick target/repo interactively:
  - `thesa`
  - `thesa --tui`
  - `thesa -T`
- Clone one repo:
  - `thesa torvalds/linux --output ./repos`
- Clone all public repos for a user/org with shallow history:
  - `thesa octocat --output ./repos --depth 1`
- Clone with explicit concurrency:
  - `thesa octocat --concurrency 8 --output ./repos`
- Archive a website:
  - `thesa archive site https://example.com --full-site --max-pages 100`
  - `thesa archive url https://example.com/docs --max-depth 2`
- Verify a thesa archive directory:
  - `thesa verify ./archives/example`
  - `thesa verify ./archives/example --json`
- Preview actions before cloning:
  - `thesa org-name --dry-run`
- Models and providers:
  - `thesa --mode models --provider hf meta-llama --dry-run`
  - `thesa --mode models --provider hf meta-llama/Llama-3.1-8B-Instruct --dry-run`
  - `thesa --mode models --provider ollama llama3 --dry-run`
  - `thesa --mode models --provider hf top --dry-run`
  - `thesa --mode models --provider hf latest --dry-run`
  - `thesa --mode models --provider ollama top --dry-run`
  - `thesa --mode models --provider ollama latest --dry-run`
  - `thesa --mode models --provider civitai top --dry-run`
  - `thesa --mode models --provider civitai latest --dry-run`
  - `thesa --mode models --provider civitai 827184 --dry-run`

## Scripts

You can compile and run with helper scripts:

- Run from repo root (interactive target prompt):
  - `./run.sh`
- Pass a target directly:
  - `./run.sh <target> [output-root] [concurrency] [extra thesa flags]`

- Compile only:
  - `./scripts/compile.sh`
- Compile and run (default concurrency `4`):
  - `./scripts/run.sh <target> [output-root] [concurrency] [extra thesa flags excluding --output]`

- Publish release to crates.io:
  - `./scripts/publish.sh --allow-dirty`
  - `./scripts/publish.sh --dry-run` (safe checks only)

`run.sh` creates an output folder from the target name (for example `owner` or,
for `owner/repo`, just `repo`) and runs clones inside that folder.

## Notes

- `--concurrency` sets the concurrent clone batch size.
- private repositories are intentionally skipped/blocked.
- Cargo packages exclude archive output, session files, logs, target directories,
  and temporary test archives. Do not put secrets in project files.
