# lexer-lang v1.0.0 — API freeze

**Stable.** v1.0.0 freezes the public API. The `Cursor` — the zero-copy scanner the
0.x line built — is now under a Semantic Versioning promise: no breaking change
before `2.0`, additions only in minor releases, and the MSRV (Rust 1.85) rising only
in a minor.

## What is lexer-lang?

The scanner that turns source text into a token stream. Its one type, `Cursor`, is a
zero-copy cursor over a `&str` — the primitive every lexer is built on, hand-written
or generated. A lexer drives the cursor with peek/advance steps and turns each
scanned run into a `Token<K>`, where `K` is the language's own kind. The cursor owns
scanning and nothing else: it does not define a language's keywords (that is `K`,
from `token-lang`), collect diagnostics (the lexer author does, from the spans the
cursor reports), or store sources (that is `source-lang`).

```rust
use intern_lang::Interner;
use token_lang::{Symbol, TokenKind};
use lexer_lang::Cursor;

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Kind { Ident(Symbol), Number, Whitespace, Eof }
impl TokenKind for Kind {
    fn is_trivia(&self) -> bool { matches!(self, Kind::Whitespace) }
    fn is_eof(&self) -> bool { matches!(self, Kind::Eof) }
    fn symbol(&self) -> Option<Symbol> {
        match self { Kind::Ident(s) => Some(*s), _ => None }
    }
}

let mut interner = Interner::new();
let mut cursor = Cursor::new("x42 7");
while let Some(c) = cursor.first() {
    let kind = if c.is_whitespace() {
        cursor.eat_while(char::is_whitespace);
        Kind::Whitespace
    } else if c.is_ascii_digit() {
        cursor.eat_while(|c| c.is_ascii_digit());
        Kind::Number
    } else {
        cursor.eat_while(|c| c.is_ascii_alphanumeric());
        Kind::Ident(cursor.intern_lexeme(&mut interner))
    };
    let _token = cursor.emit(kind);
}
```

## The frozen surface

`Cursor<'a>`, grouped by what a lexer's loop does:

- **Construct** — `new`, `with_base`, `for_source` (spans in a `source-lang` map's
  global position space).
- **Look ahead** — `first`, `second` (two-character lookahead), `is_eof`,
  `remaining`, `pos`.
- **Consume** — `bump`, `bump_if`, `eat_while`.
- **Read the run** — `lexeme`, `token_span`, `intern_lexeme`.
- **Finish a token** — `emit` (keep) or `reset_token` (drop trivia).
- **Re-exports** — `Token`, `TokenKind` (`token-lang`), `Interner`, `Symbol`
  (`intern-lang`), `BytePos`, `Span` (`span-lang`), `SourceFile` (`source-lang`), so a
  lexer names only `lexer-lang`.

The full reference, including the SemVer promise, is in
[`docs/API.md`](https://github.com/jamesgober/lexer-lang/blob/main/docs/API.md).

## What changed since 0.2.0

The freeze is mostly the commitment, not new surface, but the polish pass added two
things worth locking in before `1.0`:

- **`reset_token`** — so a lexer can *discard* trivia (whitespace, comments) instead
  of being forced to emit it as tokens. Both styles — emit-trivia and drop-trivia —
  are now first-class.
- **Saturating positions** — `pos` and `token_span` saturate rather than wrap if a
  user-supplied base plus the source length exceeds the `u32` envelope, so a span is
  never silently off by `2^32`.

Also: the `token-lang` dependency moved to its frozen `1`.

## Guarantees

Property-tested over arbitrary UTF-8, so multi-byte characters and empty input are
exercised, not assumed:

- **UTF-8 advance** — `bump` returns the next `char` and advances the position by
  exactly its UTF-8 length; positions are monotonic and end at `text.len()`.
- **Never panics, never mis-slices** — on any `&str`, the cursor stays on `char`
  boundaries and a lexeme is always a valid sub-`&str`.
- **`first` / `is_eof` track the remaining text** at every step.
- **Tiling** — lexing a whole source covers `[0, len)` with no gap or overlap;
  slicing by the token spans reproduces the input.
- **Base offsetting** — a base shifts every position and span by exactly the base.
- **Allocation-free** — the cursor borrows its source for its whole life.
- **`no_std`** (no `alloc` needed); `#![forbid(unsafe_code)]`; no
  `unwrap`/`expect`/`todo!` in shipping code.

## A note on diagnostics

The cursor does not depend on `diag-lang` and never builds diagnostics — that is the
lexer author's job, and a one-liner: build a `diag_lang::Diagnostic` from the span
the cursor reports. Because `for_source` puts spans in the map's global space, the
label resolves directly. This separation is now part of the frozen design; an
integration test drives it end to end (an unterminated string becomes a rendered
diagnostic).

## Breaking changes

**None against the 0.x line's behaviour.** `reset_token` and the saturating
arithmetic are additive. From here, breaking changes require `2.0`.

## Verification

Run on Windows x86_64 (Rust stable and 1.85 MSRV); the same commands run across the
CI matrix on Linux, macOS, and Windows:

```bash
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
cargo build --no-default-features
cargo +1.85 test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo bench --no-run
cargo deny check
```

All green. Counts (default features): 16 unit + 5 property + 5 integration tests and
17 doctests. The `--no-default-features` build is clean as `no_std`.

## Installation

```toml
[dependencies]
lexer-lang = "1"
```

MSRV: Rust 1.85.

## Documentation

- [README](https://github.com/jamesgober/lexer-lang/blob/main/README.md)
- [API Reference](https://github.com/jamesgober/lexer-lang/blob/main/docs/API.md)
- [CHANGELOG](https://github.com/jamesgober/lexer-lang/blob/main/CHANGELOG.md)

---

**Full diff:** [`v0.2.0...v1.0.0`](https://github.com/jamesgober/lexer-lang/compare/v0.2.0...v1.0.0).
**Changelog:** [`CHANGELOG.md`](https://github.com/jamesgober/lexer-lang/blob/main/CHANGELOG.md#100---2026-06-28).
