# Parser technical debt overview

This document describes structural duplication and architectural gaps in `sysml-v2-parser` as of June 2026. It complements:

- **[`PARSER_BACKLOG_ROADMAP.md`](./PARSER_BACKLOG_ROADMAP.md)** — **single backlog** for all open work (start here for priorities)
- [`SYSML_V2_COMPLIANCE_GAP.md`](./SYSML_V2_COMPLIANCE_GAP.md) — what is implemented vs partial vs permissive
- [`BNF_COMPLIANCE_MATRIX.md`](./BNF_COMPLIANCE_MATRIX.md) — compact grammar-family snapshot

The parser currently passes `cargo test`, the full validation suite (`cargo test -- --include-ignored`), and strict library node-shape gates (`ExtendedLibraryDecl = 0`). Technical debt here is about **maintainability and grammar depth**, not about missing CI green.

## Current architecture (summary)

The codebase is in a **broad coverage, construct-specific modules** phase:

| Layer | Pattern |
|-------|---------|
| Top-level defs | ~25 `*_def` entry points (`item_def`, `connection_def`, `port_def`, …) |
| Package dispatch | Large ordered `if let Ok` chain in `package_body_element` (~50 branches) |
| Bodies | Per-family parsers; many still use `skip_until_brace_end` for inner content |
| Fallback | `KermlSemanticDecl`, `KermlFeatureDecl`, `ExtendedLibraryDecl` when no dedicated path matches |

That layout delivered green validation and drove `ExtendedLibraryDecl` to zero in library gates. The trade-off is **grammar unity** for **incremental delivery**.

A recent example: library declarations such as `abstract connection name : Type[multiplicity] nonunique :> redefines { ... }` require skipping a **typed header** before subclassification. When `parse_optional_definition_specialization` replaced `take_until_terminator` after `identification` without handling `: Type ... :>`, several defs failed and fell through to `ExtendedLibraryDecl`. The fix was `parse_optional_definition_header_after_identification` in [`src/parser/specialization.rs`](../src/parser/specialization.rs) — a small shared primitive, not a full rewrite.

## Where duplication appears

### 1. Definition prefix boilerplate — **P1 done (June 2026)**

[`src/parser/definition_prefix.rs`](../src/parser/definition_prefix.rs) provides `parse_definition_prefix` with `DefinitionPrefixOptions` (`DefKeywordMode`, `VisibilityPrefix`, `AnnotationMode`, optional `second_keyword` for `use case`). Migrated `*_def` parsers: item, individual, interface, metadata, connection, constraint, port, requirement, state, occurrence, flow, allocation, case / analysis / verification, view / viewpoint / rendering, use case, enum, action.

**Still on local preludes (intentional):** `part_def` (usage disambiguation), `*_usage`, `alias_def`, `dependency`, `calc_def`, `attribute_def`.

### 2. Body terminators — **structured body loop started (P2, June 2026)**

[`src/parser/body.rs`](../src/parser/body.rs) exports `parse_structured_brace_members` and `semicolon_or_structured_definition_body`. **Attribute**, **occurrence definition**, and **rendering definition** brace bodies now parse structured member nodes with recovery instead of opaque `skip_until_brace_end`.

**Structured generic bodies (June 2026):** flow, allocation, and metadata definitions/usages — doc members plus statement-skip recovery into `DefinitionBody::Brace { elements }`.

**Part/port bodies (June 2026):** part def/usage bodies retain structured member AST; port def/usage use `parse_structured_brace_members` with `PortBody::Brace { elements }` (nested ports, in/out, doc, recovery).

**Still local or opaque:** deep action/state/requirement body members beyond definition-level structured loops; alias/import paths; connect bodies in interface parsing.

**P2 (in progress):** extend structured member grammars per family beyond doc + recovery stubs.

### 3. Package dispatch (large surface, mostly intentional)

[`package_body_element`](../src/parser/package.rs) is a long ordered dispatch chain. Much of it is **disambiguation policy** (e.g. `part_def` vs `part_usage`, `attribute_def` vs `attribute_usage`). A single giant `alt(...)` is not clearly better until disambiguation rules stabilize.

**Recommended improvement (later):** sub-dispatchers grouped by keyword family (`package_body_requirement_family`, `package_body_structure_family`, …), aligned with `PACKAGE_BODY_STARTERS` in [`lex.rs`](../src/parser/lex.rs). Worth doing when adding constructs becomes painful, not preemptively.

### 4. Recovery loops (medium duplication, high value if unified)

`recover_body_element` plus `build_recovery_error_node_from_span` loops appear in `part`, `action`, `state`, `requirement`, `constraint`, `view`, and others. The shape is always: try parse member → on failure recover and skip → push `Error` node → continue.

**Recommended improvement:** `parse_structured_brace_members` in [`body.rs`](../src/parser/body.rs) is the shared entry point; migrate remaining families to family-specific `parse_one` callbacks.

### 5. AST shape duplication (structural, larger refactor)

Many `*Def` structs repeat `identification`, `specializes`, `specializes_span`, and `body`. This mirrors the compliance gap: the **shared KerML definition/usage layer** from the spec is not yet a single grammar layer in code.

**Recommended improvement (larger):** an internal `DefinitionDecl { keyword, prefixes, identification, header, body }` mapped to typed AST variants for downstream consumers. Drive this from grammar work, not from deduplication alone.

### 6. Shared usage grammar fragments — **started**

[`src/parser/usage.rs`](../src/parser/usage.rs) now centralizes small `UsageDeclaration` / `FeatureSpecializationPart` fragments: multiplicity, `TypedBy` (`:` / `defined by` / `typed by`), subsetting, and redefinition. `part_usage`, `port_usage`, `attribute_def`, `attribute_usage`, and occurrence usages have been migrated first, including `defined by` / `typed by` and multiple specialization clauses where the public AST can currently preserve them.

**Current AST caveat:** `attribute_usage` accepts extra specialization clauses for grammar coverage, but the existing public `AttributeUsage` AST only stores `typing` and `redefines`. `occurrence_usage` stores `type_name`, `subsets`, and `redefines`, using the current last-wins behavior for multiple clauses. Structured AST fidelity for `references` / `crosses` and richer body members remains a later tranche.

**Recently migrated (June 2026):** requirement/case/analysis/verification usages, action/state usages, view/rendering/viewpoint/use-case usages, and `concern_usage` route through shared `usage_header` parsing. `calc_def` uses `parse_definition_prefix`.

**Next candidates:** remaining families with local typing fragments; member-level action/state/requirement grammar (transitions, perform, satisfy, etc.).

## What is not wasteful duplication

| Pattern | Why it stays |
|---------|----------------|
| Separate modules per SysML family (`part.rs`, `requirement.rs`, …) | Clear ownership, targeted tests, incremental BNF alignment |
| Per-fixture validation tests under `tests/validation/` | Catches regressions the aggregate suite might miss |
| `ExtendedLibraryDecl` as last resort | Safety net; library gates require count = 0 on the happy path |
| Ordered dispatch in `package_body_element` | Reflects real keyword disambiguation, not arbitrary repetition |

## Relationship to compliance gaps

From [`SYSML_V2_COMPLIANCE_GAP.md`](./SYSML_V2_COMPLIANCE_GAP.md):

1. **Generic definition/usage/specialization** — still distributed across construct-specific parsers instead of one unified layer (largest architectural gap).
2. **Permissive bodies** — `skip_until_brace_end` still appears in alias, import, connect-body fallbacks, and deep behavioral body parsers; attribute/occurrence/rendering definition bodies and flow/allocation/metadata generic bodies are now structured with recovery.
3. **Expression subset** — `expr.rs` is precedence-aware but not full `OwnedExpression`.
4. **Recovery / LSP** — solid baseline; more specific diagnostics and coverage still wanted.

Duplication in code and “partial grammar” in the spec sense overlap: the same missing shared header/body grammar shows up as copy-pasted parsers *and* as `ExtendedLibraryDecl` or opaque bodies when a shortcut fails.

## Implementation plan (P1)

**Status: complete (June 2026).** Spec and checklist: [`PARSER_DEBT_P1_PLAN.md`](./PARSER_DEBT_P1_PLAN.md). Code: [`definition_prefix.rs`](../src/parser/definition_prefix.rs), [`body.rs`](../src/parser/body.rs).

## Implementation plan (P2)

**Status: complete (June 2026).** Checklist: [`PARSER_DEBT_P2_PLAN.md`](./PARSER_DEBT_P2_PLAN.md).

## Implementation plan (P3)

**Status: complete (June 2026).** Checklist: [`PARSER_DEBT_P3_PLAN.md`](./PARSER_DEBT_P3_PLAN.md).

## Implementation plan (P4)

**Status: complete (June 2026).** Checklist: [`PARSER_DEBT_P4_PLAN.md`](./PARSER_DEBT_P4_PLAN.md).

P4 delivered: internal definition headers, view/part structured bodies, usage AST fidelity (`references` / `crosses`), `implies` expression tranche, LSP recovery tests, `part.rs` module split.

## Prioritized improvements

| Priority | Change | Effort | Benefit |
|----------|--------|--------|---------|
| ~~**P1**~~ | ~~`parse_definition_prefix` + options per keyword~~ | Done | Central prelude for migrated defs |
| ~~**P1**~~ | ~~`semicolon_or_opaque_brace_body`~~ | Done | flow / allocation / metadata |
| ~~**P2**~~ | Generic structured body loop with recovery | Done (attribute/occurrence/rendering + generic flow/allocation/metadata) | Less recovery duplication; better editor behavior |
| ~~**P2**~~ | ~~Family-specific structured body members (action/state)~~ | Done (action + state via shared loop; requirement custom) | Deeper BNF members still partial |
| ~~**P2**~~ | ~~Split `package_body_element` into keyword-group sub-dispatchers~~ | Done | `try_package_body_*` in package.rs |
| **P3** | Unified definition/usage header (typing, multiplicity, subsets, redefines) | In progress; part/port/attribute/occurrence + requirement/case/action/state/view usages migrated | Spec-aligned; fixes whole classes of library edge cases |
| ~~**P3**~~ | ~~Replace `skip_until_brace_end` in high-traffic bodies~~ | In progress (connection/interface/import/alias/enumeration/usecase/constraint use `advance_to_closing_brace`) | Deeper AST; significant work per module |

## What to avoid

- **Monolithic “parser framework” rewrite** while validation and library gates are green — high risk of re-breaking `ExtendedLibraryDecl` and strict diagnostics tests.
- **Dedup-only refactors** without grammar tests — merging code paths without fixture coverage tends to hide regressions until the full library suite runs.
- **Removing fallback nodes prematurely** — keep `ExtendedLibraryDecl` at zero via dedicated parsers, not by deleting the fallback.

## Recommended workflow for refactors

1. Introduce a small shared primitive (like `parse_optional_definition_header_after_identification`).
2. Add or extend unit tests on the primitive and one representative family parser.
3. Migrate similar families in a single PR; run `cargo test -- --include-ignored`.
4. Document any family that still uses opaque bodies in this file or in module-level comments.

## Summary

| Question | Answer |
|----------|--------|
| Is there a lot of duplication? | **Yes** — especially definition prefixes, body terminators, and recovery loops. |
| Is the codebase unmaintainable? | **No** — modules and tests are coherent; debt is known and gated. |
| Best next step? | See [`PARSER_BACKLOG_ROADMAP.md`](./PARSER_BACKLOG_ROADMAP.md) § Suggested execution order. |
| Largest long-term gap? | **Unified definition/usage/specialization grammar** plus deeper body parsing, not more top-level `*_def` files. |

The validation CI regression fixed in 2026 (typed library headers after `identification`) illustrates the preferred direction: **extract shared grammar fragments** as they are discovered, keep construct modules, and let library node-shape gates enforce that dedicated parsers stay on the happy path.
