TokenKind in rustc_lexer - Rust
Description: Enum representing common lexeme types.

Skip to main content

## TokenKind

# Enum TokenKind

Source

    pub enum TokenKind {
    Show 42 variants    LineComment {
            doc_style: Option<DocStyle>,
        },
        BlockComment {
            doc_style: Option<DocStyle>,
            terminated: bool,
        },
        Whitespace,
        Frontmatter {
            has_invalid_preceding_whitespace: bool,
            invalid_infostring: bool,
        },
        Ident,
        InvalidIdent,
        RawIdent,
        UnknownPrefix,
        UnknownPrefixLifetime,
        RawLifetime,
        GuardedStrPrefix,
        Literal {
            kind: LiteralKind,
            suffix_start: u32,
        },
        Lifetime {
            starts_with_number: bool,
        },
        Semi,
        Comma,
        Dot,
        OpenParen,
        CloseParen,
        OpenBrace,
        CloseBrace,
        OpenBracket,
        CloseBracket,
        At,
        Pound,
        Tilde,
        Question,
        Colon,
        Dollar,
        Eq,
        Bang,
        Lt,
        Gt,
        Minus,
        And,
        Or,
        Plus,
        Star,
        Slash,
        Caret,
        Percent,
        Unknown,
        Eof,
    }

Expand description

Enum representing common lexeme types.

## Variants§

§

### LineComment

A line comment, e.g. `// comment`.

#### Fields

§`doc_style: Option<DocStyle>`
§

### BlockComment

A block comment, e.g. `/* block comment */`.

Block comments can be recursive, so a sequence like `/* /* */` will not be considered terminated and will result in a parsing error.

#### Fields

§`doc_style: Option<DocStyle>`
§`terminated: bool`
§

### Whitespace

Any whitespace character sequence.

§

### Frontmatter

#### Fields

§`has_invalid_preceding_whitespace: bool`
§`invalid_infostring: bool`
§

### Ident

An identifier or keyword, e.g. `ident` or `continue`.

§

### InvalidIdent

An identifier that is invalid because it contains emoji.

§

### RawIdent

A raw identifier, e.g. “r#ident”.

§

### UnknownPrefix

An unknown literal prefix, like `foo#`, `foo'`, `foo"`. Excludes literal prefixes that contain emoji, which are considered “invalid”.

Note that only the prefix (`foo`) is included in the token, not the separator (which is lexed as its own distinct token). In Rust 2021 and later, reserved prefixes are reported as errors; in earlier editions, they result in a (allowed by default) lint, and are treated as regular identifier tokens.

§

### UnknownPrefixLifetime

An unknown prefix in a lifetime, like `'foo#`.

Like `UnknownPrefix`, only the `'` and prefix are included in the token and not the separator.

§

### RawLifetime

A raw lifetime, e.g. `'r#foo`. In edition < 2021 it will be split into several tokens: `'r` and `#` and `foo`.

§

### GuardedStrPrefix

Guarded string literal prefix: `#"` or `##`.

Used for reserving “guarded strings” (RFC 3598) in edition 2024. Split into the component tokens on older editions.

§

### Literal

Literals, e.g. `12u8`, `1.0e-40`, `b"123"`. Note that `_` is an invalid suffix, but may be present here on string and float literals. Users of this type will need to check for and reject that case.

See LiteralKind for more details.

#### Fields

§`kind: LiteralKind`
§`suffix_start: u32`
§

### Lifetime

A lifetime, e.g. `'a`.

#### Fields

§`starts_with_number: bool`
§

### Semi

`;`

§

### Comma

`,`

§

### Dot

`.`

§

### OpenParen

`(`

§

### CloseParen

`)`

§

### OpenBrace

`{`

§

### CloseBrace

`}`

§

### OpenBracket

`[`

§

### CloseBracket

`]`

§

### At

`@`

§

### Pound

`#`

§

### Tilde

`~`

§

### Question

`?`

§

### Colon

`:`

§

### Dollar

`$`

§

### Eq

`=`

§

### Bang

`!`

§

### Lt

`<`

§

### Gt

`>`

§

### Minus

`-`

§

### And

`&`

§

### Or

`|`

§

### Plus

`+`

§

### Star

`*`

§

### Slash

`/`

§

### Caret

`^`

§

### Percent

`%`

§

### Unknown

Unknown token, not expected by the lexer, e.g. “№”

§

### Eof

End of input.

## Trait Implementations§

Source§

### impl Clone for TokenKind

Source§

#### fn clone(&self) -> TokenKind

Returns a duplicate of the value. Read more
1.0.0 · Source§

#### fn clone_from(&mut self, source: &Self)

Performs copy-assignment from `source`. Read more
Source§

### impl Debug for TokenKind

Source§

#### fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

### impl PartialEq for TokenKind

Source§

#### fn eq(&self, other: &TokenKind) -> bool

Tests for `self` and `other` values to be equal, and is used by `==`.
1.0.0 · Source§

#### fn ne(&self, other: &Rhs) -> bool

Tests for `!=`. The default implementation is almost always sufficient, and should not be overridden without very good reason.
Source§

### impl Copy for TokenKind

Source§

### impl Eq for TokenKind

Source§

### impl StructuralPartialEq for TokenKind

## Auto Trait Implementations§

§

### impl Freeze for TokenKind

§

### impl RefUnwindSafe for TokenKind

§

### impl Send for TokenKind

§

### impl Sync for TokenKind

§

### impl Unpin for TokenKind

§

### impl UnsafeUnpin for TokenKind

§

### impl UnwindSafe for TokenKind

## Blanket Implementations§

Source§

### impl<T> Any for T
where T: 'static + ?Sized,

Source§

#### fn type_id(&self) -> TypeId

Gets the `TypeId` of `self`. Read more
Source§

### impl<T> Borrow<T> for T
where T: ?Sized,

Source§

#### fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

### impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

#### fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

### impl<T> CloneToUninit for T
where T: Clone,

Source§

#### unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (`clone_to_uninit`)
Performs copy-assignment from `self` to `dest`. Read more
Source§

### impl<T> From<T> for T

Source§

#### fn from(t: T) -> T

Returns the argument unchanged.

Source§

### impl<T, U> Into<U> for T
where U: From<T>,

Source§

#### fn into(self) -> U

Calls `U::from(self)`.

That is, this conversion is whatever the implementation of `From<T> for U` chooses to do.

Source§

### impl<T> ToOwned for T
where T: Clone,

Source§

#### type Owned = T

The resulting type after obtaining ownership.
Source§

#### fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

#### fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

### impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

#### type Error = Infallible

The type returned in the event of a conversion error.
Source§

#### fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

### impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

#### type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

#### fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.

## Layout§

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain `repr(...)` attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.

Size: 8 bytes

Size for each variant:

- `LineComment`: 1 byte
- `BlockComment`: 2 bytes
- `Whitespace`: 0 bytes
- `Frontmatter`: 2 bytes
- `Ident`: 0 bytes
- `InvalidIdent`: 0 bytes
- `RawIdent`: 0 bytes
- `UnknownPrefix`: 0 bytes
- `UnknownPrefixLifetime`: 0 bytes
- `RawLifetime`: 0 bytes
- `GuardedStrPrefix`: 0 bytes
- `Literal`: 7 bytes
- `Lifetime`: 1 byte
- `Semi`: 0 bytes
- `Comma`: 0 bytes
- `Dot`: 0 bytes
- `OpenParen`: 0 bytes
- `CloseParen`: 0 bytes
- `OpenBrace`: 0 bytes
- `CloseBrace`: 0 bytes
- `OpenBracket`: 0 bytes
- `CloseBracket`: 0 bytes
- `At`: 0 bytes
- `Pound`: 0 bytes
- `Tilde`: 0 bytes
- `Question`: 0 bytes
- `Colon`: 0 bytes
- `Dollar`: 0 bytes
- `Eq`: 0 bytes
- `Bang`: 0 bytes
- `Lt`: 0 bytes
- `Gt`: 0 bytes
- `Minus`: 0 bytes
- `And`: 0 bytes
- `Or`: 0 bytes
- `Plus`: 0 bytes
- `Star`: 0 bytes
- `Slash`: 0 bytes
- `Caret`: 0 bytes
- `Percent`: 0 bytes
- `Unknown`: 0 bytes
- `Eof`: 0 bytes