# lexer-lang v0.2.0 — Core

The first release with domain logic, and the first MAIN-tier crate: the scanner that
turns source text into a token stream. v0.1.0 was the scaffold; v0.2.0 is the
cursor.

## What is lexer-lang?

The scanner core. Its one type, `Cursor`, is a zero-copy scanner over a `&str` — the
primitive every lexer is built on, hand-written or generated. A lexer drives the
cursor with peek/advance steps and turns each scanned run into a `Token<K>`, where
`K` is the language's own kind. The cursor owns scanning and nothing else.

## The surface

`Cursor<'a>`, grouped by what a lexer's loop does:

- **Construct** — `new`, `with_base`, and `for_source` (spans land in a
  `source-lang` map's global position space).
- **Look ahead** — `first`, `second` (two-character lookahead), `is_eof`,
  `remaining`, `pos`.
- **Consume** — `bump`, `bump_if` (for two-character operators), `eat_while` (scan a
  run).
- **Read the run** — `lexeme` (the borrowed text), `token_span` (its span),
  `intern_lexeme` (intern it to a `Symbol`).
- **Finish a token** — `emit`, which builds the `Token<K>` and starts the next.

This is where the family dependencies meet: `span-lang` for positions, `token-lang`
for the emitted token, `intern-lang` for identifier symbols, `source-lang` for the
file being scanned. The relevant types are re-exported, so a lexer names only
`lexer-lang`.

## Guarantees

Property-tested over arbitrary UTF-8, so multi-byte characters and empty input are
exercised, not assumed:

- **UTF-8 advance** — `bump` returns the next `char` and advances the position by
  exactly its UTF-8 length; positions are monotonic and end at `text.len()`.
- **Never panics, never mis-slices** — on any `&str`, the cursor stays on `char`
  boundaries and a lexeme is always a valid sub-`&str`.
- **`first` / `is_eof` track the remaining text** at every step.
- **Tiling** — lexing a whole source covers `[0, len)` with no gap or overlap;
  slicing by the token spans reproduces the input.
- **Base offsetting** — a base shifts every position and span by exactly the base and
  nothing else.
- **Allocation-free** — the cursor borrows its source for its whole life.

## A note on diagnostics

The cursor does not depend on `diag-lang` and never builds diagnostics. A lexical
error is the language's judgement: the author builds a `diag_lang::Diagnostic` from
the span the cursor reports, and because `for_source` puts spans in the map's global
space, the label resolves directly. An integration test drives this end to end — an
unterminated string becomes a rendered diagnostic. Keeping scanning total and
allocation-free is the reason; see `dev/NOTES.md`.

## Performance

Measured with Criterion on the development machine (Windows x86_64, Rust stable):

| Path | Mean |
|---|---|
| Raw scan — `bump` through a ~12 KB source | ~2.5 µs (~5 GB/s) |
| Full lex — classify, intern, emit ~6k tokens (fresh interner) | ~1.4 ms |

Raw scanning is bound by `char` iteration; a full first-pass lex is dominated by
interning fresh identifiers, not by the cursor.

## Compatibility

- **MSRV** Rust 1.85, edition 2024.
- **`no_std`** — the cursor needs neither `std` nor `alloc`. The default `std`
  feature only forwards to the family crates; `--no-default-features` keeps the whole
  surface. There is no `serde` feature — a transient cursor has nothing to serialise.
- **Breaking changes** — none against v0.1.0 (the scaffold had no surface).

## Verification

Run green on **Windows x86_64** against Rust stable and MSRV 1.85, default and all
features:

```bash
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
cargo build --no-default-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo +1.85 test --all-features
cargo bench --no-run
cargo deny check
```

Counts at this release: 13 unit + 5 property + 5 integration tests and 16 doctests.
The `--no-default-features` build is clean as `no_std`.

## What's next

The 0.x line finalizes the surface; `1.0.0` freezes it under SemVer. Deferred,
non-breaking additions are recorded in `dev/NOTES.md` — chiefly a token-stream
`Iterator` adapter over a user scan function.

## Installation

```toml
[dependencies]
lexer-lang = "0.2"
```

## Documentation

- [README](https://github.com/jamesgober/lexer-lang/blob/main/README.md)
- [API Reference](https://github.com/jamesgober/lexer-lang/blob/main/docs/API.md)
- [CHANGELOG](https://github.com/jamesgober/lexer-lang/blob/main/CHANGELOG.md)
