# cutmerge

**Cut three synchronized MCAP recordings (ego + both wrists) into merged,
per-segment clips — written in [mcap-optimiser](https://mcap.dev) byte order
with uncompressed chunks and a smart, GOP-preserving cut on the H.264 video.**

Given three time-synchronized MCAPs from a head (`ego`) camera and two wrist
cameras, plus a `*.mix.json` edit list, `cutmerge` writes one output MCAP per
clip — where **adjacent segments closer than a minute are merged** (see
[Segment merging](#segment-merging)). Each output:

- **merges** all three cameras' topics into one file, remapping the wrist
  `/camera/...` topics to `/wrist_left/...` and `/wrist_right/...` and merging
  the three `/tf_static` streams into one;
- is **cut** to the segment's `[start_ts, end_ts)` window;
- is laid out in **mcap-optimiser byte order** — single-channel chunks grouped
  by channel, every `MessageIndex` relocated into one block before the summary,
  so a cloud reader fetches a channel in one sequential range request;
- uses **uncompressed chunks** (the payloads are already H.264 / PNG, so zstd
  would only burn CPU);
- has each H.264 channel **cut on a keyframe boundary**: the leading partial GOP
  is decoded (primed from the prior keyframe) and re-encoded into a fresh,
  self-contained GOP, while every complete GOP after it passes through untouched
  — so no clip ever starts with an undecodable mid-GOP frame.

## Prerequisites

`cutmerge` links against the system **FFmpeg** libraries (libavcodec /
libavformat / libavutil / libswscale / libswresample) and **libx264** for the
smart-cut decode/re-encode, exactly like [slimcap](https://crates.io/crates/slimcap).

- **Rust** 1.85 or newer (stable) — install via [rustup](https://rustup.rs).
- **FFmpeg dev libraries + libx264** and **pkg-config**.

**macOS (Homebrew):**

```bash
brew install ffmpeg pkg-config
```

(The Homebrew `ffmpeg` bottle ships with x264 enabled.)

**Ubuntu / Debian:**

```bash
sudo apt-get install \
  libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libswresample-dev \
  libx264-dev pkg-config
```

**Optional** — the [`mcap` CLI](https://github.com/foxglove/mcap) for inspecting
outputs (`mcap info`, `mcap doctor`):

```bash
brew install mcap    # macOS; otherwise see the mcap releases page
```

## Installation

Once the [prerequisites](#prerequisites) are in place, install the latest
release from crates.io:

```bash
cargo install cutmerge
```

Or build from source in this directory:

```bash
cargo build --release
```

This produces two binaries under `target/release/`:

- `cutmerge` — the cut/merge tool
- `probe` — a verification helper (decode-check + byte-order dump)

To install both onto your `PATH` (`~/.cargo/bin`):

```bash
cargo install --path .
```

## Usage

```bash
cutmerge \
  --ego          ego.mcap \
  --wrist-left   wrist_left.mcap \
  --wrist-right  wrist_right.mcap \
  --json         clips.mix.json \
  --outdir       out
```

Writes `out/000.mcap`, `out/001.mcap`, … — one per output clip, numbered by
position (adjacent segments may be merged; see [Segment merging](#segment-merging)).

| Flag | Default | Description |
|---|---|---|
| `--ego <PATH>` | — | Head-camera MCAP. Its summary start time anchors the JSON timestamps. |
| `--wrist-left <PATH>` | — | Left-wrist MCAP. `/camera/<x>` → `/wrist_left/<x>`. |
| `--wrist-right <PATH>` | — | Right-wrist MCAP. `/camera/<x>` → `/wrist_right/<x>`. |
| `--json <PATH>` | — | Segment edit list (see below). |
| `--outdir, -o <DIR>` | — | Output directory (created if absent). |
| `--chunk-size <BYTES>` | `8388608` | Target uncompressed chunk size (8 MiB). |
| `--merge-gap-ms <MS>` | `60000` | Merge two adjacent segments into one clip when the gap between them is smaller than this (1 min) and the merged clip stays within `--max-merged-ms`. `0` disables merging. |
| `--max-merged-ms <MS>` | `300000` | Cap (5 min) on a merged clip's duration; a merge that would exceed it is not performed. |
| `--crf <N>` | `20` | x264 CRF for the leading-GOP re-encode (18 ≈ visually transparent). |
| `--preset <NAME>` | `medium` | x264 preset (`ultrafast`…`veryslow`). |
| `--library <STR>` | `slimcap/0.1.3+h264` | `library` stamped in the output header. |
| `--only-segment <N>` | — | Process only the segment with this `index` (dev iteration). |
| `--max-skew-ms <MS>` | `100` | Keep all three cameras in every clip: clamp each segment to the time span the three inputs share, basing the cut on the shortest stream (latest start / earliest end) when their coverage differs by more than this. Boundaries that agree within the tolerance are left as the edit list asked; segments with no shared overlap are skipped. |

### Edit-list format (`*.mix.json`)

A JSON array of segments; `start_ts` / `end_ts` are **milliseconds from the
recording start** (anchored on the ego file):

```json
[
  { "index": 0, "start_ts": 30000, "end_ts": 266500, "title": "…", "description": "…" },
  { "index": 1, "start_ts": 266500, "end_ts": 443000, "title": "…", "description": "…" }
]
```

### Segment merging

Before cutting, `cutmerge` coalesces **temporally-adjacent segments** so a run of
closely-spaced clips becomes one continuous clip — `segment + gap + segment →
clip`, keeping the in-between gap's data rather than dropping it. Two segments are
merged when **both** hold:

- the **gap** between them (`next.start_ts − prev.end_ts`) is **strictly less
  than `--merge-gap-ms`** (default `60000` ms = 1 min), and
- the resulting **merged span** (`next.end_ts − prev.start_ts`) is **at most
  `--max-merged-ms`** (default `300000` ms = 5 min).

Merging is greedy over segments sorted by `start_ts` and chains across a run: a
third segment folds into the growing clip only while each new gap and the running
total still satisfy both bounds. Pass `--merge-gap-ms 0` to disable merging
(every segment stays its own clip).

Output clips are numbered by position — `000.mcap`, `001.mcap`, … — so a clip's
number no longer tracks the source segment `index` once anything merges.

With the defaults, the example edit list above (gaps of 0 / 45 / 15 / 0 s;
segment 2 → 3 spans 232 s while every other adjacent pair spans > 5 min) merges
segments **2 and 3** into one clip and leaves the rest — five segments become
four clips, `000`–`003`.

### Output metadata (`metadata.json`)

Alongside the clips, `cutmerge` writes `<outdir>/metadata.json`: a JSON array with
**one object per output clip**, in output order. Each object has:

- `index` — the clip's number, matching its `<index>.mcap` file;
- `start_ts` / `end_ts` — the clip's **window** in milliseconds from the recording
  start: the requested `[start, end)` from the edit list, clamped to the shared
  three-stream overlap. They are **not** adjusted to the emitted frame times;
- `status` — one of the values below;
- `segments` — the **source edit-list entries** merged into this clip, each echoed
  **verbatim except its `index`** (every other original field preserved, original
  `start_ts` / `end_ts` intact). A clip that wasn't merged has a one-element array.

| `status` | Meaning |
|---|---|
| `success` | Written; all three streams aligned within `--max-skew-ms`, no trim. |
| `trimmed` | Written; the window was clamped to the shared overlap (skew > tolerance). |
| `dropped` | Not written; less than 15 s of overlap remained after trimming (or none at all). |
| `failed`  | The output MCAP could not be written (any error) — any partial file is removed. |

For `dropped` / `failed` entries (no clip on disk) the top-level timestamps hold
the clamped window that would have been written. `metadata.json` is emitted only
for a full run, not with `--only-segment`.

Example — two source segments merged into clip `000`, one standalone as `001`:

```json
[
  {
    "index": 0,
    "start_ts": 30000,
    "end_ts": 90000,
    "status": "success",
    "segments": [
      { "start_ts": 30000, "end_ts": 60000, "title": "…", "description": "…" },
      { "start_ts": 65000, "end_ts": 90000, "title": "…", "description": "…" }
    ]
  },
  {
    "index": 1,
    "start_ts": 200000,
    "end_ts": 230000,
    "status": "success",
    "segments": [
      { "start_ts": 200000, "end_ts": 230000, "title": "…", "description": "…" }
    ]
  }
]
```

### Verifying output

```bash
# Decode every video channel through libavcodec and confirm each clip starts on
# a keyframe and decodes with zero errors:
probe out/000.mcap

# Print the record-level byte-order (chunks grouped, one MessageIndex block,
# then the summary — the mcap-optimiser layout):
probe records out/000.mcap
```

A clean result reports `first_is_keyframe=true` and `decoded == frames` for
every `foxglove.CompressedVideo` channel, ending with
`ALL VIDEO CHANNELS DECODE CLEANLY`.

## Output channels

For the standard 3-camera rig the merged output has 16 channels, e.g.:

| Output topic | Source | Notes |
|---|---|---|
| `/ego/zed_head/...` | ego | passthrough (already namespaced) |
| `/tf` | ego | passthrough |
| `/wrist_left/image/compressed`, `/wrist_left/imu/data`, … | wrist-left | `/camera` → `/wrist_left` |
| `/wrist_right/image/compressed`, … | wrist-right | `/camera` → `/wrist_right` |
| `/tf_static` | all three | merged into one channel |

H.264 image channels (`foxglove.CompressedVideo`) get the smart cut; depth
(`sensor_msgs/CompressedImage`, PNG) and all non-video topics are cut by
timestamp and copied verbatim.

## Notes & limitations

- **Merging is time-based only.** Adjacent segments coalesce on gap and duration
  (`--merge-gap-ms` / `--max-merged-ms`) regardless of their `title` /
  `description`. When differently-labelled segments merge, `metadata.json` keeps
  every source entry (with its label) under the clip's `segments` array —
  `cutmerge` does not pick a single label — and output clips are renumbered by
  position.
- **Channel ordering** follows first-appearance in `log_time` (the slimcap
  convention), so it is input-timing dependent: whichever camera's clock starts
  first leads. It is not a fixed order across datasets.
- **`mcap doctor`** reports many "log_time less than latest" notes — these are
  *expected*: the channel-grouped optimiser layout is non-monotonic in physical
  order by design (it is read via the index, not linearly). There is no
  corruption.
- The **remap rules are specific to this rig** (ego + two wrists, `/camera` →
  `/wrist_*`, `/tf_static` merge). Adapt `src/remap.rs` for other layouts.
- Inputs must each contain a **summary section** (every recorder here produces
  one).
- The smart cut re-encodes only the **leading partial GOP** of each video
  channel per segment; if a segment's window contains no intermediate keyframe,
  that whole window is re-encoded (degrades gracefully to a full re-encode).
- Every output is clamped to the time span **all three inputs share** (see
  `--max-skew-ms`). If one camera starts late or stops early by more than the
  tolerance, the affected clips are trimmed to the shortest stream — or skipped
  when nothing overlaps — so no clip is ever missing a camera over part of its
  span. The clamp keys off each input's first/last message time, so it covers
  short recordings at the **head or tail**, not gaps in the middle of a stream.

## License

Licensed under either of

- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE))
- MIT license ([LICENSE-MIT](LICENSE-MIT))

at your option.
