# kobold-data-shim

<img src="assets/kobold_data_shim.png" width="200">


**A verifiable COBOL record-decoding shim for data-migration pipelines.** Give it a copybook and a
raw record dump; it tells you — byte-exactly — *what that COBOL record actually meant*, by composing
the oracle-proven [`gnucobol-rs`](https://github.com/infinityabundance/gnucobol-rs) compatibility
courts. Fields outside the sealed subset **fail closed** (reported `unsupported`), never silently
guessed — the reconciliation signal that real migrations need.

```text
$ kobold-record-dump --copybook CUST.cpy --record dump.bin
FIELD                      OFF  SIZE CATEGORY      VALUE                    RAW
  CUST-ID                      0     3 numeric       42                       303432
  CUST-NAME                    3     4 alphanumeric  ANNA                     414e4e41
  CUST-BAL                     7     3 numeric       -12.34                   01234d
CUST                         0    10 group         (group)
```

Every numeric value is decoded by a court proven **byte-identical to GnuCOBOL 3.2's `libcob`** under
a differential sweep. The raw bytes travel alongside every field as the audit trail.


## Operator trust layer (KOBOLD.OPERATOR.1)

Every decoded field is **accountable** — point at a value and ask *why*:

```sh
kobold-recon explain account.cpy input.dat ACCOUNT-RECORD.BALANCE --record 0 --copydir .
kobold-recon totals  account.cpy input.dat --record-len 42 --copydir .
kobold-recon decode  account.cpy input.dat --record-len 42 --dirty-mode strict
```

`explain` returns source provenance (`copybook:line`), offset/size, usage/pic, raw bytes, decoded
value, the **sealed courts** that produced it, dependent LEVEL-88s, the record hash, the explicit
non-claims, and a stale-copybook risk note. `totals` gives control totals (record count, per-field
numeric sums, condition counts, invalid/unsupported). `decode --dirty-mode` preserves dirty bytes as
evidence (or rejects in `strict`) — **never coerces**. Duplicate JSON keys are refused, not clobbered.

## Why it exists

Mainframe modernization on AWS routinely ingests VSAM/flat-file exports into S3 + Aurora/Redshift.
The single most error-prone step is *interpreting the legacy bytes*: packed decimal (COMP-3) signs,
implied decimal scale, `REDEFINES`, `OCCURS`, copybook `COPY`/`REPLACING`. A silent off-by-one in a
sign nibble or a scale is a financial discrepancy. This shim makes that step **verifiable**:

- it reuses the sealed [`gnucobol-rs`] courts — `GNURUST.2` MOVE bytes, `GNURUST.3/9` PIC (+P),
  `GNURUST.4/10` layout (+ODO), `GNURUST.5/6` COPY/REPLACING, `GNURUST.7/13` arithmetic, `GNURUST.8`
  VALUE, `GNURUST.11/12` LEVEL-88, `GNURUST.14` binary COMP/COMP-5/COMP-X, `GNURUST.15` cp500 EBCDIC
  decode, `GNURUST.16` edited-picture decode — each proven against the GnuCOBOL oracle;
- it **fails closed** on anything outside those courts, so unsupported fields are surfaced for
  reconciliation rather than mis-decoded.

## Library API

```rust
use kobold_data_shim::decode;

let copybook = "01 CUST.\n05 CUST-ID PIC 9(3).\n05 CUST-BAL PIC S9(3)V99 COMP-3.";
let record = [b'0', b'4', b'2', 0x01, 0x23, 0x4d];
let fields = decode(copybook, &record).unwrap();
assert_eq!(fields.iter().find(|f| f.name == "CUST-BAL").unwrap().value, "-12.34");
```

`decode_with_resolver(copybook, record, resolver)` expands `COPY` statements via a caller-supplied
`CopyResolver` (the CLI resolves them from `--copydir`). `decode_all(copybook, data, record_len, resolver)` is a higher-level iterator yielding one decoded
record per fixed-length record. An optional `serde` feature (off by default) derives `Serialize` on the
decoded types (values stay strings — never floats). `decode_record(...)` additionally evaluates
**LEVEL-88 condition names** (`gnucobol-rs`' `eval_88`), so a decoded record carries both fields and
condition truths.

## End-to-end reconciliation (`KOBOLD.RECON.1`)

`kobold_data_shim::recon::reconcile` (and the `kobold-recon` CLI) decode a fixed-record file into
**byte-stable** JSONL + a `kobold-recon-receipt-v1` audit + an explicit `unsupported.json`, using only
the sealed `gnucobol-rs` courts:

```json
{"record_index":0,
 "fields":{"ACCOUNT-ID":"100000","STATUS-CODE":"A","BALANCE":"5459318.55","CUST-NAME":"CUSTOMER 0000","CUST-TIER":"G","BRANCH-NO":"5895","RISK-SCORE":"796453","INTERNAL-ID":"900453110"},
 "conditions":{"ACTIVE":true,"CLOSED":false,"DELINQUENT":false,"CUST-GOLD":true},
 "audit":{"raw_offset":0,"raw_len":42,"record_sha256":"…"}}
```

(`BRANCH-NO` is `COMP`, `RISK-SCORE` is `COMP-X`, `INTERNAL-ID` is `COMP-5` — binary fields decoded
from raw storage, not text-converted.)

A committed corpus of 3 fixture families (account / payroll / insurance, 360 records, with `COPY ... REPLACING`, COMP-3, **COMP/COMP-5/COMP-X binary**, **cp500 EBCDIC text** (explicit `--encoding`, binary/packed passthrough), and alpha + numeric-range LEVEL-88s) is in [`recon/`](recon/), with a
sealing receipt at [`recon/RECEIPT-KOBOLD-RECON-1.md`](recon/RECEIPT-KOBOLD-RECON-1.md). The output is
proven byte-stable across runs and CLI == library; `unsupported.json` lists anything outside the
sealed courts — never a silent fallback. The inverse direction (`SET condition TO TRUE` →
bytes → `eval_88` true) is the [`recon/condition-set/`](recon/condition-set/) fixture.

## AWS reference architecture (S3 → verified records)

```text
Mainframe VSAM/flat export
        │  (AWS Transfer Family / Direct Connect; cp500 EBCDIC text decoded in-shim via --encoding)
        ▼
   S3 landing zone ──(ObjectCreated)──► Lambda / AWS Batch / Glue
        │                                   │ kobold-data-shim:
        │                                   │   COPY-expand copybook → lay_out → decode (cob_move/Decimal)
        │                                   │   emit Parquet/JSON  +  per-record audit (raw_hex, court, unsupported)
        ▼                                   ▼
   raw retained                     S3 (Parquet) → Athena/Glue        Aurora (txn)
                                    + S3 (audit/parity receipts) → reconciliation jobs
```

The serverless packaging (Lambda layer / container sidecar) and a benchmark harness live in sibling
repos: [`kobold-lambda-layer`](https://github.com/infinityabundance/kobold-lambda-layer) and
[`kobold-bench`](https://github.com/infinityabundance/kobold-bench).

## Parity proofs as a migration artifact

Because the kernel is oracle-proven, a migration can ship **evidence**, not assertions: the
`gnucobol-rs` differential sweeps (`FAIL=0`), Kani proofs, fuzz runs, and the machine-readable
non-claims become part of the compliance package (SOX/audits). This shim adds the per-record audit
trail (raw bytes + the court each value came from + any `unsupported` field).

## Scope & honest edges

Claims are **layered**: a court is *sealed in `gnucobol-rs`* (proven byte-exact vs the oracle) and
*composed into the KOBOLD corpus* (exercised end-to-end here). Both are stated, never conflated.

- **Composed in the KOBOLD reconciliation corpus (byte-stable, CLI == library):**
  DISPLAY / COMP-3 / **COMP / COMP-5 / COMP-X**, `PIC` widths + record offsets, fixed `OCCURS` /
  `REDEFINES (≤ target)` / `OCCURS DEPENDING ON` physical-max / `FILLER`, `COPY [REPLACING]`,
  **cp500 alphanumeric DISPLAY text with explicit `--encoding`** (binary/packed pass through untouched),
  LEVEL-88 condition predicates, and **edited-picture decode** (`GNURUST.16` — presentation string in
  JSON, oracle-proven numeric in the audit `edited` block).
- **Fails closed / future (surfaced for reconciliation, never guessed):** numeric DISPLAY under cp500
  (EBCDIC zoned sign), `COMP-6`, floats, `SET ... TO FALSE` / FALSE clause, mixed/auto-detected
  encodings, and any **business truth beyond what the copybook + sealed courts prove**.
- **Host:** little-endian ASCII (matches the `gnucobol-rs` sealed claim).

Decoded fields are **accountable**: `kobold-recon explain` shows provenance, raw bytes, the sealed
courts used, and the stale-copybook risk; `kobold-recon totals` gives control totals; `decode
--dirty-mode` preserves dirty bytes as evidence (see the operator section above).

See [`docs/enterprise-readiness.md`](docs/enterprise-readiness.md) for the SBOM / CVE / SLA stance.

## License

This crate's source is **Apache-2.0** (`LICENSE`). It links `gnucobol-rs`, which is
**LGPL-3.0-or-later** — see [`NOTICE`](NOTICE) for what that requires when you distribute a binary
(relink-ability + notice).
