# jscpd-rs Cloning Plan

## Upstream Review

The reference implementation is a TypeScript monorepo:

- `apps/jscpd` owns the CLI, option/config merging, store setup, reporters, and
  top-level execution.
- `packages/finder` discovers files, applies `.gitignore`/ignore/size/line
  filters, coordinates detection across files, and hosts most reporters.
- `packages/core` contains the detector, in-memory store, statistics, validators,
  and Rabin-Karp based clone search.
- `packages/tokenizer` maps file names/extensions to formats and tokenizes
  supported languages. Current upstream supports 223 formats and special
  block-aware tokenization for Vue, Svelte, Astro, and Markdown.

Upstream testing is conventional package-level Vitest behind `pnpm test`,
orchestrated by Turbo. The public CI workflow builds, lints, runs tests, and
then smoke-runs `./apps/jscpd/bin/jscpd ./fixtures`. Upstream does not keep a
separate public benchmark suite or a pinned set of large repositories; the
README only documents small fixture-based output-size timing/token examples and
a note about tokenizer speed on unspecified real projects.

The core flow is:

1. Parse CLI/config into options.
2. Find supported files with glob, ignore, size, line, symlink, and gitignore
   filters.
3. Tokenize each source.
4. Convert token windows of `minTokens` into hashes.
5. Use a per-format store to find matching windows and grow adjacent matches
   into clones.
6. Validate by `minLines` and optional validators.
7. Emit statistics and reports.

## MVP Scope

The first Rust MVP intentionally implements the minimum vertical slice needed to
measure whether a Rust clone has enough performance upside to continue:

- CLI with common jscpd flags.
- Partial `.jscpd.json` support.
- File discovery with the Rust `ignore` crate for `.gitignore`.
- Upstream-synchronized extension/name format registry.
- Language-agnostic non-whitespace tokenizer.
- Numeric rolling window hashing and in-memory per-format store.
- Clone growth and `minLines` validation.
- Console, consoleFull, AI, JSON, CSV, Markdown, XML PMD CPD, SARIF, badge,
  HTML, Xcode, silent, and threshold reporters.
- Benchmark script against upstream on the same target path.

Known MVP gaps:

- Tokenization is not language-compatible with upstream yet.
- The upstream format registry is synchronized, but most long-tail formats still
  use generic tokenization rather than Prism-compatible tokenization.
- `strict/mild/weak` are still converging overall. `strict` now preserves
  whitespace tokens in the native JS/TS/Oxc path and the generic tokenizer;
  `weak` strips common comment spans for generic formats.
- Terminal timing/tips/progress/verbose behavior is partially aligned with
  upstream, including clone progress and detector event output.
- Blame data is populated from native `git blame -w`. Store options currently
  match the local upstream missing-store fallback. Dynamic external stores are
  not implemented yet.
- A native Rust API now exposes detection from configured paths and prepared
  in-memory sources. Exact upstream JavaScript package API compatibility remains
  follow-up work.
- A native `jscpd-server` binary exposes the first REST surface:
  `/`, `/api/health`, `/api/stats`, `/api/check`, `/api/recheck`, and `/mcp`.
  The MCP endpoint supports initialize/session handling, core tools, and the
  statistics resource. The current snippet check reuses prepared project token
  maps after `/api/recheck`; a dedicated indexed hybrid-store path can still be
  added if server-scale benchmarks show this is needed.
- `cache`, config `listeners`, and `tokensToSkip` are parsed for option-surface
  compatibility, but upstream currently does not consume them in runtime code.
- No full parity for non-native syntax-specific token streams yet.
- Markdown front matter and fenced code blocks are extracted into embedded
  format maps, with coverage parity on the current upstream fixture.

## Growth Plan

1. Compatibility harness: run upstream and Rust on shared fixtures, compare clone
   counts, locations, statistics, reports, and exit behavior.
2. CLI/config parity: harden remaining flags, config merging rules, exit codes,
   threshold behavior, and list output.
3. Tokenizer backend: replace the MVP tokenizer with maintained crates and
   language-aware token streams. Prefer existing parsers/tokenizers over custom
   grammars.
4. Reporters: polish remaining report details and terminal UX.
5. Advanced surfaces: full non-native tokenizer parity, dynamic external
   stores, dynamic external reporters, programmatic API/server parity, and
   stricter `strict`/`mild`/`weak` parity.
6. Performance work: parallel file reads/tokenization, compact hash storage,
   faster hashers where compatible, memory profiling, and optional external
   store backends.

## Release Candidate Gate

Before publishing a release candidate, run:

```bash
scripts/release-candidate.sh
```

This wraps the strict local lint check, the default release gate, the full
coverage matrix, and the public benchmark/coverage suite into one reproducible
pre-publication command. The defaults can still be overridden with environment
variables such as `PUBLIC_CASES`, `PUBLIC_RUNS`, `PUBLIC_MIN_SPEEDUP`, and
`STRICT`.

## Current Benchmark

Command:

```bash
FORMAT=typescript RUNS=5 scripts/bench.sh jscpd/packages
```

Result on this workspace:

- Rust MVP: `0.108s` average.
- Upstream `jscpd`: `0.818s` average.
- Same file count for this run: 297 TypeScript files.

Broader command:

```bash
RUNS=3 scripts/bench.sh jscpd/packages
```

Result on this workspace:

- Rust MVP: `0.130s` average.
- Upstream `jscpd`: `0.937s` average.
- This broader run is not fully apples-to-apples yet: the MVP supports fewer
  formats than upstream.

Initial signal: continuing makes sense, but the next milestone must measure
speed while closing tokenization/report compatibility gaps.

## Compatibility Gate

The project now uses a coverage-first compatibility rule for ongoing cloning
work:

- Rust must not miss duplicated lines reported by upstream `jscpd` for the same
  file, format, input, and options.
- Rust may report additional duplicates while compatibility is converging.
- Missing upstream line coverage is a blocking compatibility failure.
- Extra Rust duplicates are tracked as diagnostics and fixed when they represent
  likely false positives or user-visible report noise.
- Exact clone pair and fragment-boundary overlap is diagnostic only: when three
  or more equivalent fragments exist, upstream and Rust may choose different
  pairs or wider/split ranges while still covering the same duplicated lines.
- Exact 1:1 parity remains a useful quality metric, but it is not the default
  gate for deciding whether the clone is viable.

Use the compatibility harness with `STRICT=coverage` to enforce this rule.
Use `scripts/compat-matrix.sh` for the current JS/TS-focused release matrix.

## Format Coverage Strategy

The format registry is generated from upstream `@jscpd/tokenizer` using
`scripts/sync-formats.mjs`. This keeps extension detection and `--list` aligned
with upstream while avoiding hand-maintained mapping drift.

Tokenizer strategy remains hybrid:

- native Rust/Oxc path for hot JS/TS formats;
- native Rust block splitting for Markdown, Vue, Svelte, and Astro embedded
  code/style/template regions;
- generic tokenizer for other recognized formats without parity claims;
- no embedded JavaScript runtime fallback. Formats that need real compatibility
  should get native Rust tokenizers and focused compat tests.

## Accepted Hard-Feature Decisions

The current release decisions are canonicalized in
`docs/release-decisions.md`. The summary below mirrors that file for quick
orientation.

These choices are part of the current cloning direction until a compatibility
gate proves they are insufficient:

- Dynamic npm reporters, stores, and plugins are post-MVP. The first release
  should implement popular built-in reporters and stores natively instead of
  embedding or casually spawning JavaScript from Rust.
- Reporter compatibility should be strict for machine-readable contracts such as
  JSON, XML, SARIF, CSV, and Markdown. HTML should stay practically compatible,
  but pixel-perfect parity is not a release blocker.
- Tokenizer compatibility stays hybrid. Use maintained Rust crates or Oxc for
  hot formats where they materially help; keep long-tail formats on generic
  tokenization until a fixture or public-repo gate shows missed upstream
  coverage.
- Node/Commander quirks are mirrored only when they are user-visible and covered
  by compatibility tests. The project should not import a JavaScript runtime to
  reproduce every incidental JS behavior.
- Blame should stay native (`git blame`/git library based) and fail per file
  where possible instead of inheriting upstream nested-repo failure modes.
- Store/cache work should stay native and demand-driven. Custom external stores
  remain documented gaps unless a real large-repo benchmark proves they are
  needed for release viability.

## Larger Local Repo Benchmarks

All commands below used TypeScript-only scanning to keep the comparison focused:

```bash
FORMAT=typescript RUNS=3 scripts/bench.sh <repo>
```

These early exploratory timings used private local repositories. The repository
names and paths are intentionally omitted; repeatable public benchmarks are
tracked separately in `docs/public-benchmark-suite.md`.

| Repo | Rust MVP | Upstream `jscpd` | Files | Rust clones | Upstream clones |
| --- | ---: | ---: | ---: | ---: | ---: |
| Private repo A | `0.447s` | `1.930s` | 316 | 93 | 475 |
| Private repo B | `0.350s` | `1.877s` | 566 Rust / 572 upstream | 371 | 1371 |
| Private repo C | `0.010s` | `0.290s` | 28 | 12 | 41 |

Broader all-format stress on private repo B:

- Rust MVP, `RUNS=2`: `0.600s` average.
- Upstream `jscpd`: first run took `70.32s`, then the benchmark was stopped.
- This broader run is not a fair compatibility comparison yet because upstream
  supports far more formats and emitted a warning for `excel-formula`.

Conclusion remains positive: the Rust path is consistently faster on larger
repos, but the next milestone must prioritize tokenizer/discovery parity before
the speedup can be treated as product-quality.

## Acceleration Pass

The first MVP was still too conservative: it used MD5 strings for token/window
hashes and prepared files sequentially. The current hot path now uses:

- zero-copy detection tokens: no per-token `String` allocation in detection;
- `xxh3_128` token hashes;
- numeric rolling window hashes instead of MD5 over concatenated strings;
- `rustc_hash::FxHashMap` for the in-memory window store;
- parallel file reads and line counting;
- parallel per-file tokenization/window preparation;
- nanosecond-resolution benchmark timing in `scripts/bench.sh`.

Updated TypeScript-only benchmark:

| Repo | Rust MVP before | Rust MVP now | Upstream `jscpd` | Approx speedup vs upstream |
| --- | ---: | ---: | ---: | ---: |
| `jscpd/packages` | `0.108s` | `0.019s` | `0.856s` | ~45x |
| Private repo A | `0.447s` | `0.085s` | `2.028s` | ~24x |
| Private repo B | `0.350s` | `0.074s` | `1.955s` | ~26x |
| Private repo C | `0.010s` | `0.009s` | `0.305s` | ~33x |

This is the first speed signal that is strong enough to justify continuing,
provided compatibility can be raised without destroying the margin.

## Core Stabilization Pass

The detector core was refactored again so later tokenizer/reporting work does
not have to revisit the detection hot path:

- introduced numeric `SourceId` and `FormatId` in the core;
- introduced `TokenStream` as the detector input contract;
- removed per-window `Frame` allocation;
- stores only first `Occurrence { source_id, token_start }` per window hash;
- streams rolling windows directly from token hashes;
- verifies matching windows on hash hits;
- shards detection by format and runs format shards in parallel.

Updated TypeScript-only benchmark after this pass:

| Repo | Rust after hash pass | Rust after core pass | Upstream `jscpd` | Approx speedup vs upstream |
| --- | ---: | ---: | ---: | ---: |
| `jscpd/packages` | `0.019s` | `0.011s` | `0.821s` | ~75x |
| Private repo A | `0.085s` | `0.038s` | `2.041s` | ~54x |
| Private repo B | `0.074s` | `0.034s` | `1.939s` | ~57x |

Do not treat these as fixed release gates yet. Before publication, choose a
small set of popular public repositories and use them as the repeatable
benchmark/compatibility suite.
