Struct syntex_syntax::tokenstream::TokenStream

pub struct TokenStream {
    // some fields omitted
}

Token Streams

TokenStreams are a syntactic abstraction over TokenTrees. The goal is for procedural macros to work over TokenStreams instead of arbitrary syntax. For now, however, we are going to cut a few corners (i.e., use some of the AST structure) when we need to for backwards compatibility. TokenStreams are collections of TokenTrees that represent a syntactic structure. The struct itself shouldn't be directly manipulated; the internal structure is not stable, and may be changed at any time in the future. The operators will not, however (except for signatures, later on).

Methods

impl TokenStream

TokenStream operators include basic destructuring, boolean operations, maybe_... operations, and maybe_..._prefix operations. Boolean operations are straightforward, indicating information about the structure of the stream. The maybe_... operations return Some<...> if the tokenstream contains the appropriate item.

Similarly, the maybe_..._prefix operations potentially return a partially-destructured stream as a pair where the first element is the expected item and the second is the remainder of the stream. As anb example,

maybe_path_prefix("a::b::c(a,b,c).foo()") -> (a::b::c, "(a,b,c).foo()")

fn mk_empty() -> TokenStream

fn from_tts(trees: Vec<TokenTree>) -> TokenStream

Convert a vector of TokenTrees into a TokenStream.

fn from_tokens(tokens: Vec<Token>) -> TokenStream

Convert a vector of Tokens into a TokenStream.

fn respan(self, span: Span) -> TokenStream

Manually change a TokenStream's span.

fn concat(left: TokenStream, right: TokenStream) -> TokenStream

Concatenates two TokenStreams into a new TokenStream.

fn is_empty(&self) -> bool

Indicate if the TokenStream is empty.

fn len(&self) -> usize

Return a TokenStream's length.

fn to_vec(&self) -> Vec<&TokenTree>

Convert a TokenStream into a vector of borrowed TokenTrees.

fn to_tts(&self) -> Vec<TokenTree>

Convert a TokenStream into a vector of TokenTrees (by cloning the TokenTrees). (This operation is an O(n) deep copy of the underlying structure.)

fn span(&self) -> Span

Return the TokenStream's span.

fn iter<'a>(&self) -> Iter

Returns an iterator over a TokenStream (as a sequence of TokenTrees).

fn split<P>(&self, pred: P) -> Split<P> where P: FnMut(&TokenTree) -> bool

Splits a TokenStream based on the provided &TokenTree -> bool predicate.

fn slice(&self, range: Range<usize>) -> TokenStream

Produce a slice of the input TokenStream from the from index, inclusive, to the to index, non-inclusive.

fn slice_from(&self, from: RangeFrom<usize>) -> TokenStream

Slice starting at the provided index, inclusive.

fn slice_to(&self, to: RangeTo<usize>) -> TokenStream

Slice up to the provided index, non-inclusive.

fn is_delimited(&self) -> bool

Indicates where the stream is a single, delimited expression (e.g., (a,b,c) or {a,b,c}).

fn maybe_delimited(&self) -> Option<TokenStream>

Returns the inside of the delimited term as a new TokenStream.

fn is_ident(&self) -> bool

Indicates if the stream is exactly one identifier.

fn maybe_ident(&self) -> Option<Ident>

Returns an identifier

fn eq_unspanned(&self, other: &TokenStream) -> bool

Compares two TokenStreams, checking equality without regarding span information.

fn as_delimited_stream(tts: Vec<TokenTree>, delim: DelimToken) -> TokenStream

Convert a vector of TokenTrees into a parentheses-delimited TokenStream.

Trait Implementations

impl Decodable for TokenStream

fn decode<__D: Decoder>(__arg_0: &mut __D) -> Result<TokenStream, __D::Error>

impl Encodable for TokenStream

fn encode<__S: Encoder>(&self, __arg_0: &mut __S) -> Result<(), __S::Error>

impl Hash for TokenStream

fn hash<__H: Hasher>(&self, __arg_0: &mut __H)

Feeds this value into the state given, updating the hasher as necessary.

fn hash_slice<H>(data: &[Self], state: &mut H) where H: Hasher

Feeds a slice of this type into the state provided.

impl Eq for TokenStream

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl PartialEq<TokenStream> for TokenStream

Checks if two TokenStreams are equivalent (including spans). For unspanned equality, see eq_unspanned.

fn eq(&self, other: &TokenStream) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=.

impl Display for TokenStream

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Index<usize> for TokenStream

type Output = TokenTree

The returned type after indexing

fn index(&self, index: usize) -> &TokenTree

The method for the indexing (Foo[Bar]) operation