# RunMat Lexer

This crate tokenizes MATLAB/Octave source code into a stream of tokens for the parser.
It uses the `logos` library to define a fast, zero-copy DFA with a small amount of
context via `LexerExtras` to handle MATLAB-specific ambiguities.

## Design goals

- Correct tokenization for the full MATLAB language surface
- Minimal, explicit state for disambiguation (apostrophe transpose vs string, section markers, etc.)
- Compatibility with the rest of the toolchain (parser, HIR, interpreter, JIT)
- Predictable tokens: avoid over-encoding semantics at the lexing stage

## Context-aware lexing

We track two pieces of context in `LexerExtras`:

- `last_was_value: bool` — true if the previous emitted token forms a value.
  Used to disambiguate `'` as transpose vs string start.
- `line_start: bool` — true if we are at the beginning of a logical line.
  Used for `%%` section markers.

## Tokens overview

- Keywords: `function if elseif else for while break continue return end`
- Additional keywords: `switch case otherwise try catch global persistent true false`
- OOP keywords: `classdef properties methods events enumeration arguments`
- Import: `import`
- Identifiers: `[A-Za-z_][A-Za-z0-9_]*`
- Numbers: integers and floats with optional exponents
- Strings:
  - Single-quoted character arrays: `'...'` with doubled quotes `''` inside
  - Double-quoted string scalars: `"..."` with doubled quotes `""` inside
- Operators and punctuation:
  - Arithmetic: `+ - * / \ ^`
  - Element-wise: `.* ./ .\ .^`
  - Relational: `== ~= < <= > >=`
  - Logical: `&& || & | ~`
  - Transpose: `'` (contextual)
  - Colon: `:`
  - Dotted member access: `.`
  - Function handle/anonymous: `@`
  - Meta-class query: `?` (e.g., `?MyClass`)
  - Assignment and separators: `= , ;`
  - Grouping and containers: `() [] {}`
- Comments & layout:
  - Line comment: `%` to end of line
  - Section marker: `%%` at start of line
  - Block comment: `%{ ... %}` (non-nesting)
  - Line continuation: `...` (skips remainder of physical line)
  - Newlines reset `line_start`

## Notable disambiguations

- Apostrophe `'`:
  - If previous token was a value (identifier, number, `) ] }`), emit `Transpose`
  - Otherwise, let the string regex capture a full single-quoted character array
- Section `%%`:
  - Only emitted when `line_start == true`; otherwise `%` starts a normal line comment
- Line continuation `...`:
  - Emits `Ellipsis` and consumes the remainder of the physical line, including any `%` comment following it

## Non-goals at lexing time

The lexer purposefully does not encode high-level semantics:
- Integer class names like `int8`/`uint64` are identifiers
- Special variables like `varargin`/`varargout`/`ans` are identifiers
- OOP features (`handle` inheritance, method attributes) are parsed/handled later
- Command/function syntax duality is resolved in parsing/semantic phases

## Tests

See `tests/` for comprehensive coverage, organized by topic:
- `lexer.rs`: core tokens, operators, single-quoted strings, comments, ellipsis
- `transpose.rs`: detailed diagnostics and assertions for apostrophe (`'`) transpose cases
- `comments_continuation.rs`: `%` line comments, `%{...%}` block comments, `%%` section markers, `...` continuation
- `operators.rs`: logical and element-wise operators (e.g., `.* ./ .\ .^ && || & | ~`)
- `namespaces.rs`: `import` paths (including wildcard) and metaclass `?ClassName`
- `oop_tokens.rs`: OOP keywords (`classdef`, `properties`, `methods`, `events`, `enumeration`, `arguments`) and function handles `@`
- `strings_chars.rs`: double-quoted string scalars and apostrophe disambiguation exercises
- `tokens_basic.rs`: identifiers, numbers, separators (`; ,`), and simple keyword smoke tests

All lexer tests pass when running the crate tests on their own.

## Guidelines for extending the lexer

- Prefer adding new tokens only when lexical distinctions are required.
- When in doubt, keep ambiguous terms as identifiers and resolve in the parser.
- If you need context to disambiguate, add a boolean/flag in `LexerExtras` and
  use a Logos callback to `Emit` or `Skip` appropriately.
- Keep regular expressions simple (no look-around) and rely on token priority
  and callbacks for precedence and control.

## Known compatibility notes

- Non-conjugate transpose `.'` is tokenized as `Dot` then `Transpose`.
  The parser should interpret this pair as the non-conjugating transpose.
- Block comments `%{...%}` are treated as non-nesting by design.
- Error-recovery is implemented to keep producing useful tokens after invalid input; in recovery mode
  double-quoted strings are recognized as a single `Str` token, while malformed single-quoted sequences may
  be split to allow downstream error reporting.

## Remaining edges

- Apostrophe vs string: extreme adjacency cases across `...` continuation and `%` comments are covered by tests; a few rare permutations may still be added as seeds (parser semantics unaffected).
- Block comments are intentionally non-nesting; any future change would be a parser/runtime decision, not lexing.
- Command-form is resolved in the parser; lexer's role is complete for milestone.

## Crate integration

- This crate only produces tokens; it does not attempt to validate grammar.
- Downstream crates (`runmat-parser`, `runmat-hir`, `runmat-ignition`, `runmat-turbine`) are responsible for structure and semantics.
