# ic-query 0.1 Design: Snapshot Cache And Complete SNS Collections

## Status

- Status: active planning note
- Release line: `0.1`
- Decision style: cache architecture and safety boundary
- Primary implementation target: reusable local snapshot cache for expensive IC
  query collections
- Current state: NNS component caches exist; SNS list/info/token/neurons use
  live bounded queries and do not yet have complete collection snapshots

## 1. Summary

`ic-query` needs a consistent way to handle queries that are cheap when bounded
but expensive or misleading when sorted, aggregated, or scanned across many
pages. SNS governance neurons are the first clear example: a single
`list_neurons(limit = 25)` query is safe, but a command such as "top neurons by
stake" only makes sense after collecting every page for that SNS.

0.1 introduces a reusable snapshot cache model for these expensive read-only
collections. A command must not cache arbitrary CLI slices such as
`--sort stake --limit 25`. Instead, refresh code materializes named data
collections, records provenance and completeness metadata, and atomically
publishes only complete snapshots.

The first intended collection is:

```text
sns/ic/<root-principal>/neurons/full
```

The same model should later support SNS proposals, SNS token metadata, deployed
SNS inventory, and NNS collections that benefit from common cache plumbing.

## 2. Problem

IC query APIs often expose paged views. A local command may need sorted or
aggregated output over the whole collection, but the source API may only return
one page at a time and may not support caller-selected ordering.

If `ic-query` silently fetches all pages during an ordinary display command, it
can create three problems:

1. It can issue many API calls without the operator realizing it.
2. One failed page can leave ambiguous partial local state.
3. Sorting a partial page can look authoritative while only covering the
   returned slice.

0.1 must make these states explicit.

## 3. Goals

1. Keep bounded live commands cheap and predictable.
2. Add a reusable snapshot cache abstraction for expensive query collections.
3. Preserve the invariant that published cache files are absent or complete.
4. Keep failed refresh attempts separate from complete snapshots.
5. Record enough provenance to explain where cached data came from.
6. Keep JSON files as the first storage backend.
7. Design storage keys and cache APIs so a later SQLite backend can be added
   without changing command semantics.
8. Reuse existing atomic-write and refresh-lock behavior where possible.
9. Prevent commands from presenting partial data as complete global rankings.

## 4. Non-Goals

0.1 should not:

1. Add SQLite as a dependency unless JSON snapshots prove inadequate.
2. Make every command cache-backed.
3. Treat cached IC metadata as deployment authority.
4. Add mutating IC operations.
5. Guarantee a source-canister point-in-time snapshot if the source API does
   not provide that guarantee.
6. Hide unbounded paging behind ordinary list commands.

## 5. Cache Model

The cache unit is a named collection snapshot, not a rendered command result.

Command options such as `--format`, `--verbose`, `--sort`, and display
`--limit` are view options. They must not affect the snapshot key.

Snapshot keys should be logical before they are paths:

```text
domain: sns | nns
network: ic
entity: root principal | component name
collection: neurons | token | deployed-snses | subnets | nodes
scope: full | owner:<principal>
```

The JSON backend can encode that key as paths:

```text
.icq/sns/ic/deployed-snses.json
.icq/sns/ic/<root-principal>/token.json
.icq/sns/ic/<root-principal>/neurons/full.json
.icq/sns/ic/<root-principal>/neurons/owner-<principal>.json
```

The path layout is an implementation detail of the JSON backend. Command code
should depend on logical cache keys and snapshot structs.

## 6. Snapshot Shape

Every published snapshot should include:

```text
schema_version
network
source_endpoint
fetched_at_unix_secs
domain
entity
collection
scope
completeness
data
```

For SNS neurons, a first JSON shape can be:

```json
{
  "schema_version": 1,
  "network": "ic",
  "source_endpoint": "https://icp-api.io",
  "fetched_at_unix_secs": 1781360000,
  "domain": "sns",
  "entity": {
    "root_canister_id": "example-root",
    "governance_canister_id": "example-governance",
    "name": "Example SNS"
  },
  "collection": "neurons",
  "scope": {
    "kind": "full"
  },
  "completeness": {
    "status": "api_exhausted",
    "page_size": 100,
    "page_count": 12,
    "row_count": 1142,
    "point_in_time_guaranteed": false
  },
  "data": {
    "neurons": []
  }
}
```

Unknown future schema versions must be rejected unless a migration exists.

## 7. Complete Refresh Contract

Published snapshot files must be transactional:

```text
.icq/.../neurons/full.json
```

is always either absent or complete.

Refresh attempts use separate files:

```text
.icq/.../neurons/full.refresh.lock
.icq/.../neurons/full.refresh-attempt.json
```

Refresh flow:

1. Acquire the refresh lock.
2. Create or replace the attempt file with `status = "running"`.
3. Fetch pages sequentially or with a deliberately bounded strategy.
4. After each successful page, update attempt progress.
5. If a page fails, record `status = "failed"` and preserve the previous
   complete snapshot.
6. If the API is exhausted, write a complete snapshot to a temp file.
7. Atomically replace the published snapshot.
8. Release the refresh lock.

Normal read commands must not treat refresh-attempt files as authoritative.

## 8. Paging And API Load Policy

Live commands may use bounded requests:

```text
icq sns neurons 1 --limit 25
```

Commands that require whole-collection semantics must use complete snapshots:

```text
icq sns neurons 1 --sort stake
icq sns neurons 1 --top stake
```

If the complete snapshot is missing, the command may either:

1. print the same style of notice used by existing caches and refresh/create
   the cache; or
2. fail with a clear instruction to run a refresh command.

The command must not silently perform unbounded paging without a visible cache
refresh path.

Refresh commands should expose safety controls:

```text
--page-size <count>
--max-pages <count>
--source-endpoint <url>
--refresh-lock-stale-after <duration>
```

If `--max-pages` stops the scan before API exhaustion, the result is an
incomplete attempt and must not replace the complete snapshot.

## 9. Reusable Interfaces

The storage layer should separate logical snapshot behavior from JSON files:

```text
SnapshotKey
SnapshotEnvelope<T>
SnapshotStore
PagedCollector
RefreshAttempt
RefreshOutcome
```

Suggested responsibilities:

```text
SnapshotStore:
  load_latest(key)
  write_complete_snapshot(key, envelope)
  with_refresh_lock(key, stale_after, operation)
  write_refresh_attempt(key, attempt)

PagedCollector:
  fetch_page(cursor, page_size)
```

SNS neurons then become one `PagedCollector` implementation. The generic
refresh helper owns locks, attempt state, retries, atomic commit, and
complete/incomplete semantics.

## 10. SQLite Transition

The first backend should remain JSON because it matches the current repository,
is easy to inspect, and avoids native dependency complexity.

The logical key model should make a future SQLite backend straightforward:

```text
cache_snapshots(
  id,
  domain,
  network,
  entity,
  collection,
  scope,
  schema_version,
  fetched_at_unix_secs,
  completeness_status
)

sns_neurons(
  snapshot_id,
  neuron_id,
  cached_neuron_stake_e8s,
  maturity_e8s_equivalent,
  staked_maturity_e8s_equivalent,
  created_timestamp_seconds
)
```

JSON atomic rename maps to a SQLite transaction:

1. Insert a refresh attempt.
2. Insert rows associated with that attempt.
3. Mark the attempt committed.
4. Make it the latest complete snapshot in the same transaction.

Command behavior should not change when the backend changes.

## 11. SNS Neuron Command Direction

The current live command remains useful for quick inspection:

```text
icq sns neurons <id|root-principal> --limit 25
```

The cache-backed extension should add an explicit refresh surface:

```text
icq sns neurons refresh <id|root-principal>
```

After a complete snapshot exists, display commands can support deterministic
client-side views:

```text
icq sns neurons <id|root-principal> --sort id
icq sns neurons <id|root-principal> --sort stake
icq sns neurons <id|root-principal> --sort created
```

If a sort requires a complete collection and only a live bounded page is
available, `ic-query` must say so.

## 12. Validation Plan

0.1 cache work should include:

```text
cargo fmt --all -- --check
cargo clippy --all-targets --all-features --locked -- -D warnings
cargo test --all-targets --all-features --locked
git diff --check
```

Focused tests should prove:

1. Missing cache behavior is explicit.
2. Complete snapshots load and render.
3. Unknown schema versions are rejected.
4. Failed refresh attempts preserve the previous complete snapshot.
5. Refresh locks prevent concurrent writers.
6. Stale locks can be reclaimed by policy.
7. Partial page scans do not publish complete snapshots.
8. Client-side global sorts require complete snapshots.
9. JSON output includes completeness and provenance metadata.

## 13. Exit Criteria

0.1 is complete when:

1. A reusable snapshot cache module exists.
2. JSON is the default snapshot backend.
3. SNS neurons can be refreshed into a complete local snapshot.
4. Failed SNS neuron refreshes preserve the previous complete snapshot.
5. Sorts that require full SNS neuron collections only run against complete
   snapshots.
6. Cache docs and command help clearly describe live versus cached behavior.
7. The design status file records implementation deltas and remaining risks.
