Lex-Babel Format Implementation Guide

This document provides the standard workflow for implementing format interoperability in lex-babel.
It covers project setup, implementation approach, testing strategy, and best practices.

For brevity, we list spec files from the root spec directory (specs/v1).

Before you start, read the v1 interop scope:

    The authoritative tiering — which formats are Core, Stretch, Experimental, Planned, or
    explicitly out of scope — lives in `comms/docs/interop-scope.lex`. Read it first. The short
    version: Markdown (both ways), HTML export, PDF export, and PNG export are Core. HTML
    import is Stretch. RFC XML import is Experimental. Pandoc and LaTeX are Planned (not
    started). PDF import is a category error and will not be implemented, ever. This guide
    assumes you're working inside that scope.

Read for reference:
    lex-babel/src/format.rs           # Format trait definition
    lex-babel/src/lib.rs              # Architecture overview
    lex-babel/src/ir/                 # Intermediate Representation
    lex-babel/src/common/           # Nested ↔ Flat conversion algorithms

The rationale is to start with export (Lex → Format), since we start with a well-understood AST
(lex-parser's), making the first task easier. Import (Format → Lex) comes second, leveraging
the export implementation for testing.

0. Setup and Dependencies

  0.1. Add format-specific dependencies to Cargo.toml:

    Example for Markdown:
      [dependencies]
      lex-parser = { path = "../lex-parser" }
      comrak = "0.29"  # Markdown parser/serializer

    For other formats, choose appropriate well-maintained crates:
      - HTML: Use a robust HTML parser like `html5ever` or `scraper`
      - Pandoc (planned, not started — see interop-scope §5): the plan is `pandoc_ast`,
        not shelling out
      - LaTeX (planned, export only via Pandoc — see interop-scope §5): no bespoke
        LaTeX library; the path is `lex → Pandoc AST → LaTeX`

  0.2. File structure to create:

    src/formats/<format>/
      ├── mod.rs           # Module documentation, Format trait impl, element mapping table
      ├── serializer.rs    # Lex AST → Format (export)
      ├── parser.rs        # Format → Lex AST (import)
      └── mapping.rs       # Shared conversion utilities (optional, add if needed)

  0.3. Register format in src/formats/mod.rs:

    pub mod <format>;
    pub use <format>::<Format>Format;

  0.4. Register in src/registry.rs (if not auto-discovered)

  0.5. Add test module:

    tests/
      ├── lib.rs           # Add `mod <format>;`
      └── <format>/
          ├── mod.rs       # Test suite entry point
          ├── export.rs    # Export tests (Lex → Format)
          ├── import.rs    # Import tests (Format → Lex)
          └── fixtures/    # Test files (.lex and .<ext> pairs)

  0.6. Element Mapping Table (maintain in src/formats/<format>/mod.rs as doc comment):

    Create a complete mapping table with these columns:

    | Lex Element | Format Equivalent | Export Notes | Import Notes |

    Include all IR node types:
    - Document, Heading, Paragraph, List, ListItem, Definition, Verbatim, Annotation
    - InlineContent: Text, Bold, Italic, Code, Math, Reference

    Document lossy conversions explicitly.
    See src/formats/markdown/mod.rs for a complete example.

  0.6.1. Special Case: Links and Anchors

    Lex does not have link anchors (clickable text). It only has references like [url].
    When converting to/from formats that DO support links with anchors (HTML, Markdown, etc.),
    follow this standard behavior:

    EXPORT (Lex → Format with links):
      - The word BEFORE the reference becomes the link anchor
      - If the reference is the first word in an element, use the word AFTER it
      - The reference itself becomes the link href/target

      Example:
        Lex:      "welcome to the bahamas [bahamas.gov]"
        Markdown: "welcome to the [bahamas](bahamas.gov)"
        HTML:     "welcome to the <a href=\"bahamas.gov\">bahamas</a>"

      Example (reference at start):
        Lex:      "[wikipedia.org] Wikipedia is useful"
        Markdown: "[Wikipedia](wikipedia.org) is useful"

    IMPORT (Format with links → Lex):
      - The anchor text becomes plain text in the content
      - The reference follows it in Lex format

      Example:
        HTML:     "<a href=\"bahamas.gov\">bahamas</a>"
        Lex:      "bahamas [bahamas.gov]"

    IMPLEMENTATION:
      - Generic helpers are provided in src/common/links.rs
      - Use extract_anchor_for_reference() during export
      - Use insert_reference_with_anchor() during import
      - These work at the IR InlineContent level, so they're format-agnostic

  0.7. Reference Documents for Testing:

    Identify a canonical reference document for the format:
    - Markdown: CommonMark specification examples
    - HTML: HTML5 test suite snippets
    - LaTeX: Common LaTeX document patterns

    Create: tests/fixtures/<format>-reference.<ext> with representative examples
    - Markdown currently snapshots:
        * tests/fixtures/markdown-reference-commonmark.md (CommonMark spec README)
        * tests/fixtures/markdown-reference-comrak.md (Comrak README)
      These are imported in lex-babel/tests/markdown/import.rs and asserted via insta snapshots.
      Verify round-trips from the CLI with:
        cargo run --bin lex -- lex-babel/tests/fixtures/markdown-reference-commonmark.md --to tag

1. What to test: as a general rule we want to have:

  Getting the right source files for testing and iterating is key. 


  1.1. Isolated unitests , one per element (in some cases 2 or 3 might be useful). These serve as a simplified way to get to know the other ast and verify correctness.
  1.2 Test the trickiest bit: the document structure, being Lex hierarchical while most formats are flat. For this, we will use the trifecta documents (./docs/spec/v1/trifecta), starting rom the simplest case, then an intermediate (files 010, 020, and 060). 
  1.3 Ensambles: documents that mix many elemenents. We always test:
    1.3.1 Kitchensink: specs/v1/benchmark/010-kitchensink.lex botn as import and export since it's a reasonable concide document that covers all Lex format featus.
    1.3.3 Other format reference document: most format's tooling / library we will use have a reference file  , we must find such a file and use it for each format.

  That is: 1-3 files per elment (according to complexity), 3 trifecta structure files, the full kitchensink document and a reference file for that format itself.


2. Implementation Approach

  General pipeline for all formats:

    Export: Lex AST → IR → Events → Format AST → String
    Import: String → Format AST → Events → IR → Lex AST

  2.1. Export (Lex → Format) Implementation:

    Step 1: Convert Lex AST to IR
      - Use existing `lex_babel::to_ir(doc)` function
      - This gives you IR::Document with IR nodes
      - IMPORTANT: Annotations are attached as metadata to AST nodes, not stored in the content tree
      - You must extract annotations from each element using element.annotations()
      - See lex-parser/src/lex/assembling/stages/attach_annotations.rs for how they're attached
      - See lex-babel/src/ir/from_lex.rs:extract_attached_annotations() for extraction example

    Step 2: Convert IR to flat event stream
      - Use `common::nested_to_flat::tree_to_events(&ir_doc_node)`
      - This flattens the hierarchical structure into start/end events

    Step 3: Convert events to Format AST (format-specific library)
      - Walk events and build format-specific nodes
      - Handle format constraints (e.g., heading depth limits)
      - You will need state machines to manage conversion (see section 2.3 below)

    Step 4: Serialize Format AST to string
      - Use format library's serializer
      - Configure output options as needed
      - Post-process to clean up library artifacts (e.g., unwanted HTML comments)
      - Some libraries inject elements you may not want in the final output

    Example skeleton for serializer.rs:

      use crate::ir::events::Event;
      use crate::error::FormatError;

      pub fn serialize_to_<format>(doc: &lex_parser::lex::ast::Document) -> Result<String, FormatError> {
          // Step 1: Lex AST → IR
          let ir_doc = crate::to_ir(doc);

          // Step 2: IR → Events
          let events = crate::common::nested_to_flat::tree_to_events(&ir_doc);

          // Step 3: Events → Format AST (format-specific)
          let format_ast = events_to_<format>_ast(&events)?;

          // Step 4: Format AST → String (using format library)
          let output = <format_library>::serialize(format_ast)?;
          Ok(output)
      }

  2.2. Import (Format → Lex) Implementation:

    Step 1: Parse format string to Format AST
      - Use format library's parser
      - Handle parse errors gracefully

    Step 2: Convert Format AST to IR events
      - Walk format AST nodes recursively
      - Emit Event::Start*, Event::Inline, Event::End* appropriately
      - Maintain proper nesting with start/end pairs

    Step 3: Convert events to IR tree
      - Use `common::flat_to_nested::events_to_tree(&events)`
      - This reconstructs the hierarchical structure
      - Validates proper nesting

    Step 4: Convert IR to Lex AST
      - Use existing `ir::to_lex::to_lex_document(&ir_doc)` (implement if missing)
      - Map IR nodes to Lex AST elements

    For format-specific implementation examples, see:
      - src/formats/markdown/mod.rs (once implemented)
      - src/formats/tag/mod.rs (simpler example, export only)

  2.3. Common State Machines for Event ↔ Format AST Conversion:

    When converting between event streams and format-specific ASTs, you will need
    several state machines to manage the conversion process. These patterns recur
    across all format implementations:

    2.3.1. Parent Stack (for building trees from events)
      - Maintains the current nesting level during AST construction
      - Push containers onto stack when entering (StartHeading, StartList, etc.)
      - Pop when exiting (EndHeading, EndList, etc.)
      - Current parent is always stack.last_mut()
      - Example: src/formats/markdown/serializer.rs:66-68

    2.3.2. Heading Hierarchy (handled automatically by flat_to_nested)
      - The generic flat_to_nested converter automatically closes parent headings
      - Format parsers just emit StartHeading(level) - no EndHeading needed
      - When flat_to_nested sees a new heading, it auto-closes any at same or deeper level
      - At document end, it auto-closes all remaining open headings
      - This is built into src/common/flat_to_nested.rs - all formats get it for free
      - Example: src/formats/markdown/parser.rs just emits StartHeading

    2.3.3. Content Accumulation (for multi-event elements)
      - Collect content across multiple inline events before finalizing
      - Common for verbatim blocks (accumulate lines) and definitions
      - Use temporary buffers: in_verbatim flag + verbatim_content string
      - Example: src/formats/markdown/serializer.rs:70-72, 199-221

    2.3.4. Multi-Element Pattern Detection (for complex common)
      - Some Lex elements map to patterns across multiple format elements
      - Example: Lex Definition → Markdown "**Term**: description" + siblings
      - Requires lookahead/peekable iterators to consume sibling nodes
      - Track boundaries (headings, other definitions) to know when to stop
      - Example: src/formats/markdown/parser.rs:360-413

    2.3.5. Context-Sensitive Rendering (handling format quirks)
      - List items may need auto-wrapped paragraphs (Markdown tight lists)
      - Headings can only contain inline content, not blocks
      - Use flags: current_heading, in_list_item, list_item_paragraph
      - Example: src/formats/markdown/serializer.rs:77-82, 114-130, 244-260

    These state machines interact and must be carefully coordinated. When in doubt,
    study the markdown implementation as a reference for managing these patterns.

3. How To Test, for each direction (import and export)

  The backbone of testing will be comprised of unit tests testing Lex ↔ Format AST conversions.
  The base format for tests follows, and is to be followed by isolated document tests and the trifecta tests.

    3.1. Unit test pattern (AST level):

      use lex_parser::lex::transforms::standard::STRING_TO_AST;

      #[test]
      fn test_paragraph_export() {
          // Load Lex source from spec file
          let lex_src = std::fs::read_to_string("../../specs/v1/elements/paragraph.docs/paragraph-01-flat-oneline.lex").unwrap();
          let lex_doc = STRING_TO_AST.run(lex_src).unwrap();

          // Optionally test IR conversion
          let ir = lex_babel::to_ir(&lex_doc);
          let events = tree_to_events(&DocNode::Document(ir));

          // Assert on event structure
          assert!(matches!(events[1], Event::StartParagraph));
          assert!(matches!(events[2], Event::Inline(_)));

          // Full string conversion
          let output = <Format>Format.serialize(&lex_doc).unwrap();
          assert!(output.contains("expected content"));
      }

    3.2. Integration test pattern (string level):

      #[test]
      fn test_kitchensink_roundtrip() {
          let lex_src = std::fs::read_to_string("tests/fixtures/kitchensink.lex").unwrap();
          let lex_doc = LexFormat.parse(&lex_src).unwrap();

          // Export to format
          let format_output = <Format>Format.serialize(&lex_doc).unwrap();

          // Import back to Lex
          let lex_doc2 = <Format>Format.parse(&format_output).unwrap();

          // Compare (may be lossy, so use snapshot testing)
          // Use `insta` crate for snapshot testing
          insta::assert_snapshot!(format_output);
      }

  Note: Use lex-parser's STRING_TO_AST.run() or similar parsing utilities to load
  isolated elements from spec files for testing.

    3.3. Snapshot Testing Best Practices:

      Snapshot tests are superior to assertion-based tests for format conversion because
      they capture the exact output and make regressions immediately visible.

      Setup:
        $ cargo install cargo-insta

      Workflow:
        1. Write test using insta::assert_snapshot!(output)
        2. Run test - it will fail on first run (no snapshot exists)
        3. Review generated snapshot in snapshots/ directory
        4. If correct: cargo insta accept
        5. If incorrect: fix code and rerun

      Best practices:
        - Review snapshot diffs carefully before accepting changes
        - Document lossy conversions in snapshots (e.g., annotations not reimported)
        - Use snapshot names that match test function names for clarity
        - Commit snapshots to git alongside code
        - Snapshots catch regressions better than hand-written assertions
        - Use insta::assert_snapshot!(output, @"expected") for inline snapshots

      Example from markdown implementation:
        - See lex-babel/tests/markdown/export.rs for snapshot test usage
        - See lex-babel/tests/markdown/snapshots/ for snapshot files


4. Testing Order

  1. Always start with Lex export to format (easier since you control the input AST).

  2. Do the isolated elements, one by one, testing and committing every element that is working.

     General suggested order (adapt to format):
     - Paragraph (simplest, no nesting)
     - Heading (maps to Session)
     - Bold, Italic, Code inlines
     - Lists (introduces nesting)
     - Code blocks / Verbatim
     - Definitions (may need creative mapping)
     - Annotations (often lossy conversion)

  3. Do the trifecta documents, one by one. It's key to get each right in order (start from smaller file numbers).
     Trying to get the high number files working first won't work, you need to walk up the complexity ladder.

     Trifecta test files (in /specs/v1/trifecta/):
     - 010-paragraphs-sessions-flat-single.lex (simplest)
     - 020-paragraphs-sessions-flat-multiple.lex
     - 060-trifecta-nesting.lex (most complex nesting)

  4. Do the benchmark files, again, one by one, starting with the kitchensink.

     Benchmark files:
     - specs/v1/benchmark/010-kitchensink.lex (export)
     - tests/fixtures/<format>-reference.<ext> (import)

  5. Additional cases that are relevant for that format, be it in isolated element form or full ensembles.

  Once you start the work on the import, by using the same documents, you can leverage the export
  tests: you are now sure of one AST form (Lex from export tests), and the other format's AST
  (the library for the format is understood to be correct), which makes testing conversion much easier.

5. Remember:

  1. While format specific transformations exist, there is an entire class of mapping challenges that
     will be true across formats. The classic and most complex one is the mapping from nested documents
     (Lex) to flat documents (Markdown, HTML, Pandoc, LaTeX). This is why we have
     lex-babel/src/common/mod.rs, that is base source code that handles these. If you find new
     common that affect multiple formats, add to the common library for reuse.

  2. Keep as cargo document, in the formats module file (i.e. lex-babel/src/formats/<format>/mod.rs)
     a table of ported elements on import and export columns.

  3. As soon as feasible add your format to lex-cli so you can quickly test the work on the command line.
     You can add both text output (<format>) and AST output (<format>-ast) to make intermediate steps easier.

6. Tips:

  Useful commands during development:

    $ cargo run --bin lex inspect <path> ast-treeviz  # Visualize Lex AST
    $ cargo run --bin lex convert <path> to <path>    # Test conversion
    $ cargo test -- --nocapture                       # Run tests with output
    $ cargo test <format>                             # Run format-specific tests
