# mcpify fixes to apply before regenerating bamboo-mcp-rs

Two independent generator-level fixes, both to land in mcpify and then be picked
up by regenerating `bamboo-mcp-rs` from scratch (per your call — no hand-patch
track for either). A third topic (CI/CD workflow generation) turned out to
need no code change at all — see the note at the end.

## Fix 1: Compress generated_schemas.json to unblock crates.io publish

### Context

`bamboo-mcp-rs` was generated by `mcpify` (source at `/Users/lucianoguerche/Documents/GitHub/mcpify`) from a Bamboo (Atlassian) OpenAPI spec. `cargo publish --dry-run` reported the package at ~14 MiB compressed — over crates.io's 10 MiB cap — almost entirely because of `src/validation/generated_schemas.json`, a 191 MB JSON file embedded via `include_str!` in `validator.rs:32`.

Root cause, confirmed by direct inspection: every one of the 404 operations' `inputSchema`/`outputSchema` documents embeds a full, byte-identical copy of the OpenAPI spec's entire component-schema library (`$defs`, ~143 KB) — 808 copies ≈ 115 MB of pure duplication. This happens in mcpify's `schema_resolve.rs`, and is a deliberate design choice (see the comment at lines 37-42: fully standalone per-operation documents are "simpler and more robust than cross-schema `$ref` resolution, at the cost of some file size").

We considered and ruled out:
- **Deduping `$defs` into one shared block per file** — works, but requires every target's validator to re-expand `$defs` before validating (a semantic change, 5x over), and reopens the "simpler/more robust" tradeoff mcpify's author deliberately chose.
- **Splitting into multiple files ("multi-volume")** — doesn't help: crates.io's cap is on total packaged (tar.gz'd) size, not any single file, so spreading the same bytes across more files doesn't reduce the total.
- **Protobuf/binary schema format** — the content is arbitrary, self-describing JSON Schema, not a fixed message shape. Protobuf would fall back to a generic `Struct`/`Value`, buying no real size/type-safety win, while risking silent `int64→double` precision loss on round-trip (this API has `format: int64` fields) and adding protobuf toolchain/runtime dependencies to all 5 target languages.
- **Plain gzip** — only gets to ~13.9 MB (still over the cap), because gzip's 32 KB LZ77 window is smaller than the 143 KB repeat distance, so it can't eliminate the duplication.

**Chosen fix:** compress the existing (unchanged) JSON with a **long-distance-matching compressor (zstd, `--long`)**. Measured on the actual file: level 19 + `--long=27` → **44 KB** (4s to compress, ~30ms to decompress); level 3 → 66 KB in 0.14s. This eliminates the duplication automatically, requires zero changes to `schema_resolve.rs`'s JSON Schema semantics, and leaves ~150-200x headroom under the cap. No dedup, no splitting, no protobuf.

Runtime handling is the same shape across all 5 targets, since decompression is fast enough (~30ms measured) that no on-disk caching is needed anywhere:
- **Rust**: embeds the asset at compile time via `include_str!`. Switch to `include_bytes!` on the compressed `.zst` file; decompress once into memory per process (existing `OnceLock` cache).
- **TypeScript/Python/Go/C#**: already read the asset from disk at runtime. Read the `.zst` file's bytes and decompress directly into memory (existing per-target in-memory cache — `lru_cache`/module-level/static, unchanged), same as today just with a decompression step added. No decompressed `.json` sibling file is ever written to disk — avoids read-only-filesystem failure modes and stale-cache-after-upgrade concerns entirely.

This fix lands entirely in mcpify itself; `bamboo-mcp-rs` will be regenerated from scratch afterward rather than hand-patched, so there is no separate track for it.

### mcpify generator changes (Fix 1)

**Shared writer** — `src/schemas_asset.rs`, `write_schemas_json_at()` (lines 21-47): after building the JSON `Value` as today, `serde_json::to_vec()` it, zstd-compress (level ~19, `--long=27` or similar window) via the `zstd` crate, and write the compressed bytes to `<path>.zst` instead of writing JSON text to `<path>`. Add `zstd` to mcpify's own `Cargo.toml` (`[dependencies]`, currently no compression crate present). This function is already shared by all 5 targets and by `add-version`, so this one change covers every writer path.

**Filename plumbing** — anywhere a `schemas_file` path is recorded or derived needs its extension updated to `.json.zst`:
- Each target's schema-asset-path constant, e.g. `targets/rust/steps/tools.rs:34`'s `GENERATED_SCHEMAS_PATH`, and the equivalent constant in `targets/{typescript,python,go,csharp}/steps/tools.rs`.
- `add_version/mod.rs`'s `sibling_path()` (lines 173-188), which derives version-suffixed names (e.g. `generated_schemas_v11.2.json`) from the default path — verify/adjust its suffix-insertion logic handles the double extension correctly (insert `_v<label>` before `.json.zst`, not before just `.zst`).

**Per-target validator templates** — same mechanical shape in each, no JSON Schema semantics touched:
- `targets/rust/templates/validation/validator.rs.tera`: `schemas_json_for` returns `&'static [u8]` via `include_bytes!`; `schemas_by_operation` decompresses with `zstd::decode_all(bytes)` into a `String`, then `serde_json::from_str` as today. Update the header comment (lines 1-11) to mention compression. Mirror the matching codegen in `targets/rust/steps/versions.rs`'s `validator_body()` (lines 69-81): emit `include_bytes!("...json.zst")` instead of `include_str!("...json")`.
- `targets/{typescript,python,go,csharp}/templates/validation/validator.*.tera` (representative: `python/templates/validation/validator.py.tera`, lines 32-40 `_schemas_by_operation`): read the `.zst` file's bytes instead of the plain `.json` text, decompress in memory, then `json.loads`/`JSON.parse` as today — same in-memory cache (`lru_cache` etc.), no disk writes added.
- Each target's dependency manifest template needs a zstd decompression library added: Go → `github.com/klauspost/compress/zstd` (pure Go); Python → `zstandard`; C# → `ZstdSharp.Port`; TypeScript/Node → a maintained zstd package (no built-in zstd support to rely on across supported Node versions).

**Test fixtures**: `schemas_asset.rs`'s own tests (lines 49-110) assert on plain JSON text read back from disk — update to decompress before asserting. Same for `python/templates/tests/unit/validation/test_validator.py.tera` and `csharp/templates/Tests/Validation/ValidatorTests.cs.tera` if they hardcode the old shape — check when implementing.

## Fix 2: Load mcp_store*.db into memory to avoid held file locks

### Context

You recalled that with a previous mcpify-generated project (jira-dc-mcp), trying to update the SQLite store file while the MCP server was running failed with "database is locked" — the running process's open file handle/lock on the `.db` file blocks external replacement or a concurrent write (e.g. re-running `populate-embeddings`).

In bamboo-mcp-rs, every read call site — `core/mcp_server.rs:91,103,124` (the `search`/`get`/`call` tool handlers) and `cli/get.rs:9`, `cli/search.rs:9`, `cli/call.rs:15` — calls `open_store()` (`data/store.rs:55-59`), which opens a **fresh, file-backed, read-only** connection every time. Only `bin/populate_embeddings.rs:35` uses `open_store_read_write` against the real on-disk file — that one's the actual writer and is unaffected by this fix.

Important existing constraint to preserve: the `call` handler's comment at `core/mcp_server.rs:120-122` explains the connection is deliberately opened and dropped *before* any `.await`, because `rusqlite::Connection` isn't `Sync` and holding a `&Connection` across an await point would make the tool-call future non-`Send`. Any caching we add must keep respecting that — a shared connection must live behind a `Mutex`, and callers must still finish with it (drop the guard) before awaiting anything.

### Design

Add a cache, mirroring the pattern `validation/validator.rs` already uses for schemas (`OnceLock` keyed by `api_version`): on first access for a given version, open the on-disk file **read-only**, copy its entire contents into a fresh `Connection::open_in_memory()` via `rusqlite`'s backup API (`rusqlite::backup::Backup::new(&disk_conn, &mut mem_conn)?.run_to_completion(...)`), then **drop the disk connection immediately** — its file handle/lock is held only for the brief copy, not for the process's lifetime. Cache the resulting in-memory `Connection` (wrapped in a `Mutex`, `Box::leak`'d to `'static`, same shape as the existing schema cache) keyed by `api_version`.

Callers replace `open_store(&db_path)` with something like `store_connection(&api_version)?.lock().unwrap()`, use the guard, and drop it before awaiting — same discipline already documented at `core/mcp_server.rs:120-122`, just applied to a shared cached connection instead of a fresh per-call one. `mcp_server.rs`'s `Self { db_path, .. }` field likely becomes `api_version: String` instead, matching how `validate_input`/`validate_output` already take `api_version` and resolve their own resource rather than a pre-resolved path — a small but consistent architectural alignment.

`sqlite-vec`'s virtual table (`semantic_endpoints`) is unaffected: `register_vec_extension()` uses `sqlite3_auto_extension`, which applies process-wide to every connection including in-memory ones, and the backup API operates below the SQL layer so it faithfully copies the vec0 virtual table's shadow tables.

Since this pattern is portable — every one of the 5 target languages' SQLite bindings has an equivalent (Python's stdlib `sqlite3.Connection.backup()`, Go bindings' backup support, C#'s `SqliteConnection.BackupDatabase()`, Node's binding of choice) — apply the same "open once, back up to memory, cache, never re-touch the disk file" shape to all 5 targets' data-access templates, not just Rust's.

### Where this lands in mcpify

- `targets/rust/templates/data/store.rs.tera` (mirrors `data/store.rs` exactly today, confirmed via direct comparison) — add the backup-to-memory + per-version cache, keep `open_store_read_write` untouched for the `populate-embeddings` writer path.
- `targets/rust/templates/core/mcp_server.rs.tera` and the `cli/{get,search,call}.rs.tera` templates — swap call sites to the new cached accessor, threading `api_version` instead of a resolved `db_path` where needed.
- Equivalent data-access templates in `targets/{typescript,python,go,csharp}` — same shape, using each language's own backup-to-memory mechanism (verify exact API/feature-flag needs per language during implementation — e.g. confirm whether rusqlite's `backup` module needs an explicit Cargo feature or is available by default).

## Note: CI/CD workflow generation — no work needed

mcpify already generates `ci.yml`, `docker-build.yml`, and `release.yml` for all 5 targets via `generate_enterprise_scaffolding`, gated by the existing `--publish-registry` CLI flag (`cli.rs:51-54`). bamboo-mcp-rs's current hand-written workflows exist because the original `mcpify generate` invocation didn't pass that flag (consistent with the original `Cargo.toml` having `publish = false` and `.github/` being untracked). When you regenerate, just pass `--publish-registry` — you'll get a correct `release.yml` (including the binary-attachment step your hand-written version is missing) already paired with Fix 1's compression, so its `cargo publish` step will actually succeed. You'll still want to re-apply the CI matrix customization (macOS+Linux, for sqlite-vec's native extension) and re-add `perf.yml` (a bespoke diagnostic workflow with no mcpify template) by hand afterward.

## Verification

- In mcpify: run its existing test suite (`schemas_asset.rs`, `schema_resolve.rs`, `data/store`-equivalent tests — updating any fixtures that assert on the old plain-JSON shape or the old per-call-open connection pattern) after the template changes.
- Generate a fresh throwaway project per target (or at minimum the Rust target) from a real spec to confirm: the compressed schema asset is produced and validated correctly end-to-end, and the store loads once into memory with no lingering open handle on the `.db` file after startup (e.g. confirm the on-disk file can be replaced/deleted while the generated server is still running).
- Regenerate `bamboo-mcp-rs` from scratch with the fixed mcpify (passing `--publish-registry`), then in that project: `cargo test`, `cargo publish --dry-run` (confirm package size is now comfortably under 10 MiB), and a functional smoke test — run the MCP server and call `search`/`get`/`call` end-to-end to confirm both schema validation and store reads behave correctly, and that `populate-embeddings` can still run (and, ideally, that it can run concurrently with a live server without hitting "database is locked").

## Outcome

Both fixes were implemented across all 5 targets (Rust, TypeScript, Python, Go, C#) and validated: mcpify's full unit suite (321 tests) passes; real e2e generation + build + test runs passed for Rust, TypeScript, and C# (which caught and fixed a genuine C# compile error — field initializers can't reference instance members, so the cached in-memory connection uses a manual lazy-init pattern instead of `Lazy<T>`); Go's build and lint passed, with test execution blocked only by a missing native ONNX library unrelated to these changes; Python's `aiosqlite.backup()` usage was verified directly against the installed package source after its e2e test hit a slow first-time model-download timeout.
