Lexer

The implementation of the _lexer is contained within this mixin template.

To use it, this template should be mixed in to a struct that represents the _lexer for your language. This struct should implement the following methods:

popFront, which should call this mixin's _popFront() and additionally perform any token filtering or shuffling you deem necessary. For example, you can implement popFront to skip comment or tokens.
A function that serves as the default token lexing function. For most languages this will be the identifier lexing function. This should then be passed to the Lexer template mixin as the template parameter.
A function that is able to determine if an identifier/keyword has come to an end. This function must return $(D_KEYWORD bool) and take a single $(D_KEYWORD size_t) argument representing the number of bytes to skip over before looking for a separating character.
Any functions referred to in the tokenHandlers template paramater. These functions must be marked $(D_KEYWORD pure nothrow), take no arguments, and return a token
A constructor that initializes the range field as well as calls popFront() exactly once (to initialize the _front field).

mixin template Lexer (

Token

alias defaultTokenFunction

alias tokenSeparatingFunction

alias staticTokens

alias dynamicTokens

alias possibleDefaultTokens

alias tokenHandlers

) {

const(Token) front();

enum tokenSearch;

}

Members

Functions

_popFront void _popFront(): Advances the lexer to the next token and stores the new current token in the _front variable.
empty bool empty(): Implements the range primitive empty.
front const(Token) front(): Implements the range primitive front.

Variables

_front Token _front;: The token that is currently at the front of the range.
range LexerRange range;: The lexer input.

Parameters

Token: TokenStructure
defaultTokenFunction: defaultTokenFunction
tokenSeparatingFunction: tokenSeparatingFunction
staticTokens: staticTokens
dynamicTokens: dynamicTokens
possibleDefaultTokens: possibleDefaultTokens
tokenHandlers: tokenHandlers

Examples

1 struct CalculatorLexer
2 {
3     mixin Lexer!(IdType, Token, defaultTokenFunction, isSeparating,
4         staticTokens, dynamicTokens, possibleDefaultTokens, tokenHandlers);
5 
6     this (ubyte[] bytes)
7     {
8         this.range = LexerRange(bytes);
9         popFront();
10     }
11 
12     void popFront() pure
13     {
14         _popFront();
15     }
16 
17     Token lexNumber() pure nothrow @safe
18     {
19         // implementation goes here
20     }
21 
22     Token lexWhitespace() pure nothrow @safe
23     {
24         // implementation goes here
25     }
26 
27     Token defaultTokenFunction() pure nothrow @safe
28     {
29         // There is no default token in the example calculator language, so
30         // this is always an error.
31         range.popFront();
32         return Token(tok!"");
33     }
34 
35     bool isSeparating(size_t offset) pure nothrow @safe
36     {
37         // For this example language, always return true.
38         return true;
39     }
40 }