# Expose a public lexer layer beneath the datum-tree Reader (for parinfer-style consumers)

lispexp's public surface is two layers over the same dialect `Options`: a **Lexer** that turns source into a linear token stream tiling the input (delimiters with their `Delim`, atoms, strings, line/block comments, char literals, reader-syntax markers, whitespace — each with a span), and a **Reader** that builds the `Parsed` datum tree on top of it. This is driven by a second use case beyond `cccc`: serving as the backend for a parinfer implementation. Studying `parinfer-rust-emacs` showed parinfer never builds a tree — it runs a character-level lexical state machine (`In::Code`/`String`/`Comment`/`LispBlockComment{depth}`/`GuileBlockComment`/`JanetLongString`…) with per-dialect feature flags (`lisp_block_comments`, `guile_block_comments`, `scheme_sexp_comments`, `janet_long_strings`, `lisp_vline_symbols`, `string_delimiters`) that map almost 1:1 onto lispexp's Options, tracking a paren stack and indentation to re-infer parens. The lexer must therefore surface strings and comments as spans (which the tree deliberately omits, ADR-0010), report escape state (so `#\(` and `"\)"` don't miscount as delimiters), and remain robust on incomplete/unbalanced input at character granularity — it lexes linearly and never does the top-level resync that is a *parser* policy (ADR-0004). Non-goal: lispexp provides the dialect-aware lexical analysis but does **not** implement the parinfer paren-inference algorithm itself; that stays in the consumer. That parinfer independently arrived at the same orthogonal per-dialect flag model is corroboration of ADR-0003.
