# Architecture

Internal design notes for contributors and maintainers. Not load-bearing for users — see [README.md](README.md) and [docs.rs](https://docs.rs/sqpack-rs) for the public API.

## The decision: format parsers don't know about install layout

The single architectural decision this crate is built around is the **strict separation between format parsing and install-layout awareness**.

Nothing under `src/index.rs`, `src/index2.rs`, `src/header.rs`, or `src/dat.rs` knows where files live on disk, how categories are named, which archive a game path routes to, or what platform conventions look like. These modules take bytes in (via `Read + Seek`) and produce parsed structures. They are pure data — bytes in, typed values out.

Anything that needs that knowledge — install discovery, path-to-archive routing, composed read operations across multiple archives — sits **above** the format layer and consumes it. The format layer never imports from above.

The grep-verifiable invariant: no upward imports from `index.rs`, `index2.rs`, `dat.rs`. If a future addition wants to reach across the boundary, the layer below stays unchanged; the addition imports from format and adapts.

This framing tells you where new code goes:

- Parses on-disk bytes into typed structures without knowing what install they came from? → a format module.
- Composes parsed structures with install-layout knowledge (paths, categories, file discovery)? → its own top-level module above format.
- Operates over composed structures (scans, integrity checks, transformations)? → its own top-level module alongside the composition modules.

New capabilities are **siblings, not a stack**. The rule for a hypothetical `Foo` capability: it lives at `src/foo/`, not nested inside another capability. The flat module list at the top of `src/` is meaningful precisely because each name is one capability — keep it that way and the architecture stays self-explaining.

## Format — `src/header.rs`, `src/index.rs`, `src/index2.rs`, `src/dat.rs`

Pure wire-format parsers. Each module is structured the same way:

1. binrw-derived on-disk structs at the top (header, entry, block, etc.).
2. A public type (`Index`, `Index2`, `DatFile`) with an `open(path)` constructor.
3. Accessors that return decoded views of the on-disk data (e.g. `IndexEntry::dat_location`, `Index2Entry::file_hash`).
4. `#[cfg(test)] mod tests` at the bottom with hand-built byte buffers.

Format modules currently call `std::fs` directly via the `open(path)` constructor. `Index` and `Index2` additionally expose a `read<R: Read + Seek>(reader)` constructor — the seam for a filesystem abstraction; when one lands, the path-taking constructor becomes a one-line shim and the reader-taking signature is unchanged. The index parsers consume the reader fully, so the reader's lifetime stops at the constructor.

`DatFile` deliberately omits the public `read(reader)` constructor. Unlike `Index`/`Index2` it does not parse the entire file up front: it keeps a buffered `File` handle open and seeks on demand during each `read_file` / `read_file_into` call. Generalising over an arbitrary `R: Read + Seek` would force `DatFile` to either acquire a type parameter (and propagate it to every call site) or box the reader behind dynamic dispatch — neither pays for itself today. The asymmetry is intentional and is the reason the "standard module shape" line above lists only `open(path)`. A test-only `from_buffer(buf)` constructor inside `src/dat.rs` exercises the same parse logic over an in-memory cursor; if a public reader-based `DatFile` constructor is ever needed, it lands as an additive change.

Shared types live in `src/header.rs`:

- `SqPackHeader` — generic wrapper present on every archive file.
- `SqPackIndexHeader` — index-specific second header for `.index` / `.index2`.
- `SqPackFileType` — wire-level discriminant (`Dat = 1`, `Index = 2`, plus `Unknown(u32)` for forward compatibility).
- `Platform` — `Win32`/`Ps3`/`Ps4`/`Unknown(u16)` variants, mirroring `zipatch-rs::Platform`. The on-disk `platform_id` stays a `u8`; the enum's `Unknown` payload widens to `u16` so cross-crate bridges stay lossless. Used by future composition modules to resolve platform-specific path suffixes; the format layer itself doesn't care.
- `DatLocation` — `(data_file_id, byte_offset)` projection shared between v1 and v2 index entries.

## Errors — `src/error.rs`

Today: single `Error` enum covering all parse failures (`InvalidMagic`, `UnexpectedFileType`, `InvalidHeaderSize`, `DuplicateHash`, `Decode { context, message }`, `MisalignedIndexData`, `UnsupportedFileKind`, `BlockHeaderSizeMismatch`, `DecompressedSizeMismatch`, `InvalidBlockLayout`, `Io`). The `context: &'static str` field on `Decode` carries per-call-site identifiers (`"sqpack_header"`, `"index_header"`, `"index_entry"`, `"index2_entry"`, `"file_info"`, `"block_info"`, `"block_header"`).

The single-enum shape is intentional for the current scope. As the surface grows, it is expected to split into per-domain errors with an umbrella `Error` + `From` impls. Pre-1.0 that split is a minor-version bump; planning for it now keeps `Error::Decode { context }` as the parser-only variant rather than collapsing every failure mode into one shape.

## Module shape conventions (the unwritten ones)

Conventions that have emerged across the existing modules. Codified here so they stop being inferred from sibling files.

**On-disk structs use `binrw`.** `#[derive(BinRead)]` with `#[br(little)]` for the FFXIV wire format. Magic validation is explicit (not `#[br(magic = ...)]`) so `Error::InvalidMagic` stays distinct from `Error::Decode`. Reserved fields are kept on the struct (visible to binrw layout) and either `pub(crate)` with `#[allow(dead_code)]`, or `pub` when the parent type is `#[non_exhaustive]`.

**Public entry types are `#[non_exhaustive]`.** `IndexEntry`, `Index2Entry`, `FileInfo`, `BlockInfo`, `BlockHeader` all carry the attribute. Wrapper types (`Index`, `Index2`, `DatFile`) don't need it because their fields are private.

**Tracing discipline:**

- `info!` once per logical operation the consumer cares about (an archive opened, a file extracted) with counts/byte-totals/elapsed-ms.
- `debug!` per-section / per-sub-header.
- `trace!` per-entry / per-block / forensic detail in hot loops.
- `warn!` for tolerated anomalies.
- No `error!` — errors return via `Error`.
- Top-level operations use `info_span!`; sub-operations use `debug_span!`. Paths captured as `%path.display()`.

The schema is undocumented at the rustdoc level today. When the public surface stabilises, span/event/field names will be promoted to a SemVer-tracked schema in a `tracing_schema` constants module. Plan for that placement now; keep emission sites consistent so the schema crystallises cleanly.

**Validation precedes seeks.** Every `read` path validates structural bounds before any seek that depends on them. `Index::read` and `Index2::read` validate `header.size >= 24` and `index_data_offset >= header.size + index_header.size`. `DatFile::read_file_into` validates `byte_offset >= header.size`. The pattern: a corrupt header field must never let the reader seek backwards into already-parsed bytes and re-interpret them as something else.

**Reserved bits get `debug_assert!`, not silent ignore.** `Index2Entry`'s bit 0 of `data` is reserved (the synonym mechanism doesn't apply to v2's full-path CRC32 key). `read_inner` carries a `debug_assert!(data & 1 == 0)`. Documented as reserved, asserted in debug builds, tolerated in release. Real shipping files have always satisfied it; the assertion exists to catch the first one that doesn't.

## Composition — `src/hash.rs`, `src/route.rs`, `src/sqpack.rs`

Sibling modules at `src/` that sit **above** the format layer and consume it.
This is exactly the case the architectural decision at the top of this file
anticipates: the format modules stay pure and these modules add install-layout
knowledge alongside them.

`src/hash.rs` carries the FFXIV path-hashing primitives. Standard reflected
CRC32 byte loop (polynomial `0xEDB8_8320`, init `0xFFFF_FFFF`) but **without**
the standard final XOR — matching Lumina's C# `Crc32.Get` and the actual
on-disk `.index` / `.index2` hash keys. To recover the standard zlib / gzip /
PNG CRC32 of the same bytes, XOR the output with `0xFFFF_FFFF`. The byte loop
and 256-entry table are built at compile time via `const fn` — no new crate
dependency. Two public key shapes are exposed: the v1 split-on-last-slash
pair (`hash_index1_path`, `hash_index1_key`) and the v2 full-path single
CRC32 (`hash_index2`). A shared `pub(crate) fn normalize(&str) -> Vec<u8>`
performs the lowercase + slash-normalisation step so the routing layer below
uses the same byte sequence the hasher sees, and pre-normalised
`pub(crate)` variants (`hash_index1_key_normalized`, `hash_index2_normalized`)
let `SqPack::read_file` allocate the normalised buffer exactly once on its
hot path.

`src/route.rs` resolves a game-relative path to a physical archive triple.
`Category::try_from_name` carries the wire-id table (`common = 0x00`,
`bg = 0x02`, `exd = 0x0a`, …); `ArchiveTriple::from_game_path` returns the
`(category, expansion, chunk = 0)` triple. The expansion id comes from the
second path segment when it parses as `ex<N>`; the literal token `ffxiv`
does **not** bump the expansion. `repo_folder` is a `&'static str` lookup
into a 16-entry array indexed by expansion id and returns
`Option<&'static str>`; values above 15 yield `None` (no clamping — a
clamped folder would silently route to the wrong archive). Path-construction
methods (`index_path`, `index2_path`, `dat_path`) return `Option<PathBuf>`
and yield `None` for both `Platform::Unknown` and out-of-range expansions.
Chunk is always `0` from `from_game_path`; the layer above probes from
chunk 0 upward and stops once a chunk has no `.index` / `.index2` on disk.

`src/sqpack.rs` is the one-call entry point. `SqPack::open` validates that
`<install>/sqpack/` exists; the `with_platform` builder rejects
`Platform::Unknown` so downstream lookups can rely on the path-builders
returning `Some`. Nothing else is touched until `read_file`
runs. The lookup flow chains hashing, routing, `Index`/`Index2` open, and
`DatFile` read in that order — see the module rustdoc for the per-step
error mapping. The chunk probe walks from 0 upward; the early-break
condition (neither `.index` nor `.index2` exists for the current chunk)
is the actual correctness control, and `CHUNK_PROBE_CAP = 7` is a
defensive safety belt against a malformed install. The v0 contract is
deliberately lazy: no archive pre-scanning, no inter-call caching,
archives re-opened per call. Eager caching is a follow-up and is much
easier to add than to retract once consumers depend on cache-invalidation
semantics.

The new `Error` variants — `UnknownCategory`, `PathNotFound`,
`UnsupportedPlatform` — all belong to this composition layer, which is
also the only layer that knows the concept of "game path" exists. The
format modules continue to surface only structural failures. The legacy
`SynonymUnsupported` variant is retained on the enum for SemVer
compatibility but is no longer produced: v1 synonym-flagged entries
resolve through the parsed [`Index::resolve_synonym`](crate::Index::resolve_synonym)
table at this layer, returning a real [`DatLocation`](crate::DatLocation)
on a hit or falling through to v2 / the next chunk on a miss.

`src/verify.rs` adds the persisted-hash capability one rung above the
composition modules above. It consumes `SqPack` / `Category` / `FileKind`
but never imports from `dat.rs` / `index.rs` / `index2.rs` directly —
the hash pipeline is built on `SqPack::read_file_streaming` +
`SqPackFileStream::read_block_into`, with one reused scratch buffer per
builder / verifier so peak memory is bounded by one block rather than
one full file. Manifest + builder + verifier live together rather than
splitting into `manifest.rs` + `verify.rs` because the types form a
tightly coupled set (the manifest is the verifier's input shape, the
builder is the manifest's only non-trivial constructor, and
`MismatchKind::Unverifiable` carries the same `FileKind` the builder
surfaces via `ManifestAddOutcome::Skipped`); if a future capability
diverges materially from verification, the split becomes additive.
The `serde` Cargo feature gates `Serialize` / `Deserialize` derives on
`SqPackManifest` and `ManifestEntry` only — the verifier itself works
without the feature; default-off so consumers that don't persist
manifests don't pull `serde` into their dep graph. `sha2` is an
always-on runtime dependency — the verifier is the primary motivation
for the crate's repair-flow consumer, so gating it behind a feature
would force every downstream caller to opt in to the headline
capability.

The same module also carries the hash-keyed manifest sibling
(`SqPackHashManifest` / `SqPackHashManifestBuilder` /
`SqPackHashVerifier`). Where `SqPackManifest` requires the caller to
supply a path list ("verify these specific files"), the hash-keyed
variant is keyed on `IndexHashKey` — the same hash the on-disk
`.index` / `.index2` entries use — so the builder can sweep an entire
install via `SqPack::enumerate_archives` and hash every standard-kind
entry without an external path list. The two manifests coexist because
they serve different jobs: path-keyed for readable repair flows
(curated path lists, human-readable mismatches), hash-keyed for
"clean install once, verify forever" baselines that surface drift
without curation. The hash-keyed verifier also reports `extras` — entries
present in the live install but not in the manifest — which the
path-keyed flow can't see by construction. `serde` derives are gated
on the hash-keyed types too; the runtime hash pipeline is the same
`SHA-256 + read_block_into` reuse as the path-keyed builder.

## Operational siblings — `src/cancel.rs`, `src/observer.rs`, `src/tracing_schema.rs`

Three more siblings at `src/` carry the verifier's progress / cancel /
tracing surface. They sit alongside `verify.rs` rather than nested
inside it: a `CancelToken` belongs to the consumer's thread topology,
not to the verifier's scan state, and the observer trait is shaped
around the verifier's `Mismatch` output today. New scanning
capabilities (the manifest builder's progress hooks, the archive
walker's progress reporting) will either adapt to the existing trait
surface or extend it with new methods — observers stay in this sibling
module rather than splitting per-capability, the same way
`zipatch-rs::ApplyObserver` is a single shared trait across apply /
verify / plan-build there.

`src/cancel.rs` carries `CancelToken` — a cloneable
`Arc<AtomicBool>` wrapper installed via
`SqPackVerifier::with_cancel_token`. Verbatim shape lift from
`zipatch-rs::apply::cancel`; the only differences are namespace
references in the rustdoc and the partial-report exit pattern. A
cancelled scan returns a `VerifyReport` with `cancelled = true` rather
than the `ApplyError::Cancelled` shape `zipatch-rs` uses — verification's
"collect every issue you can find" goal makes a hard-stop error worse
than a graceful partial-report exit. The streaming variant
(`VerifyStream`) mirrors this: `next()` returns `None` once cancellation
is observed, while `cancelled() / verified_count() / skipped_count()`
expose work-done-so-far.

`src/observer.rs` carries the `SqPackObserver` trait + `NoopObserver`
default. Per-file callbacks (`on_file_verified` / `on_mismatch` /
`on_skipped`) plus a `should_cancel` poll fire at each manifest-entry
boundary inside `SqPackVerifier::verify` / `verify_streaming`. The
trait carries `Send` only (not `Sync`) — the verifier holds the
observer by exclusive borrow during the scan, so multi-reader access is
not part of the contract. Composes with `CancelToken`: both are polled
at the same point, either can abort. Lifted from
`zipatch-rs::apply::observer` and adapted for the verifier's
per-entry-callback shape; the `ApplyObserver::on_chunk_applied`
single-entry-point design doesn't carry over because verification has
three distinct outcome paths per entry, each useful to a launcher's UI.

`src/tracing_schema.rs` is the crate-internal source of truth for the
`tracing` span-name catalog promoted to a SemVer-tracked public contract
in `src/lib.rs`'s `# Tracing` section. The module is `pub(crate)`; every
emission site in the crate references the const there rather than an
inline literal so a rename lands in a single file. Field names stay as
literals at the emission sites (the `tracing` macro syntax for
const-named fields would clutter calls without a rename-cost benefit);
they're documented as part of the stable vocabulary in the `lib.rs`
catalog. Event-completion message strings stay as literals at the
emission sites — the rustdoc catalog tracks them — matching the
`zipatch-rs::tracing_schema` precedent.

## Decompression backend

DEFLATE decompression is feature-gated via the `flate2` backend triad: `rust-backend` (default, pure-Rust via miniz_oxide), `zlib-rs` (pure-Rust port of zlib), and `zlib-ng` (C-based, fastest). All three are mutually exclusive Cargo features routed through an optional `flate2` dep. A `compile_error!` guard in `src/lib.rs` rejects builds with no backend enabled.

Decompression is the hot path; the triad was added in `feature/flate2-backends` so consumers can swap backends without forking the crate.

## Repository layout

Single crate, single workspace member. No CLI subcrate today. If one is added, the convention will be a workspace member alongside, not a feature-gated `src/cli/` module inside the library — extraction is cheaper before the CLI has any features than after.

## Module map

```
src/
├── lib.rs              — crate root, re-exports, flate2 backend compile_error guard, tracing catalog
├── error.rs            — Error enum, Result alias, Error::decode(context, &err) constructor
├── header.rs           — SqPackHeader, SqPackIndexHeader, SqPackFileType, Platform, DatLocation
├── index.rs            — Index (v1, 16-byte entries, folder<<32|file composite 64-bit key)
├── index2.rs           — Index2 (v2, 8-byte entries, full-path u32 CRC32 key)
├── dat.rs              — DatFile, FileInfo, BlockInfo, BlockHeader, FileKind, Standard-payload read
├── hash.rs             — crc32 primitive, hash_index1_path / _key, hash_index2
├── route.rs            — Category, ArchiveTriple, install-relative path composition
├── sqpack.rs           — SqPack: one-call open(install_root) / read_file(game_path)
├── vfs.rs              — Vfs trait + StdFs / InMemoryFs impls (read-only fs abstraction)
├── verify.rs           — SqPackManifest, SqPackManifestBuilder, SqPackVerifier (SHA-256 integrity)
├── cancel.rs           — CancelToken (cloneable Arc<AtomicBool> wrapper for verifier aborts)
├── observer.rs         — SqPackObserver trait + NoopObserver default for verifier callbacks
└── tracing_schema.rs   — crate-internal canonical span-name consts (pub(crate))

tests/
├── common/             — maybe_data_dir() helper for skip-on-absent integration tests
├── index_real_file.rs   — #[ignore]'d, runs against a local FFXIV install
├── index2_real_file.rs  — #[ignore]'d
└── dat_real_file.rs     — #[ignore]'d
```

## Ecosystem context

`sqpack-rs` is a standalone library with no network dependency and no runtime requirements beyond Rust's standard library plus the dependencies listed in `Cargo.toml`. It is consumed by separate workspaces as a path or registry dependency.

`zipatch-rs` is a sibling library for the FFXIV ZiPatch binary patch format. The two crates have no compile- or runtime dependency on each other; consumers that need both bring them in independently. Conventions are shared between the two repos by hand — the error-context pattern, the tracing-level discipline, the `#[non_exhaustive]` rule, the `flate2` backend feature triad. When a sibling crate evolves a convention worth adopting, this doc gets updated, not the code re-derived.
