Lexer

The implementation of the _lexer is contained within this mixin template.

To use it, this template should be mixed in to a struct that represents the _lexer for your language. This struct should implement the following methods:

popFront, which should call this mixin's _popFront() and additionally perform any token filtering or shuffling you deem necessary. For example, you can implement popFront to skip comment or tokens.
A function that serves as the default token lexing function. For most languages this will be the identifier lexing function. This should then be passed to the Lexer template mixin as the template parameter.
A function that is able to determine if an identifier/keyword has come to an end. This function must return $(D_KEYWORD bool) and take a single $(D_KEYWORD size_t) argument representing the number of bytes to skip over before looking for a separating character.
Any functions referred to in the tokenHandlers template paramater. These functions must be marked $(D_KEYWORD pure nothrow), take no arguments, and return a token
A constructor that initializes the range field as well as calls popFront() exactly once (to initialize the _front field).

mixin template Lexer (

Token

alias defaultTokenFunction

alias tokenSeparatingFunction

alias staticTokens

alias dynamicTokens

alias possibleDefaultTokens

alias tokenHandlers

) {

static string generateMask(ubyte[] arr);

const(Token) front();

void _popFront();

bool empty();

static string escape(string input);

enum tokenSearch;

static ulong getFront(ubyte[] arr);

void advance(Token token);

LexerRange range;

Token _front;

}

Members

Functions

_popFront void _popFront(): Advances the lexer to the next token and stores the new current token in the _front variable.
advance void advance(Token token): Undocumented in source. Be warned that the author may not have intended to support it.
empty bool empty(): Implements the range primitive empty.
front const(Token) front(): Implements the range primitive front.

Manifest constants

tokenSearch enum tokenSearch;: Undocumented in source.

Static functions

escape string escape(string input): Undocumented in source. Be warned that the author may not have intended to support it.
generateMask string generateMask(ubyte[] arr): Undocumented in source. Be warned that the author may not have intended to support it.
getFront ulong getFront(ubyte[] arr): Undocumented in source. Be warned that the author may not have intended to support it.

Variables

_front Token _front;: The token that is currently at the front of the range.
range LexerRange range;: The lexer input.

Parameters

Token: TokenStructure
defaultTokenFunction: defaultTokenFunction
tokenSeparatingFunction: tokenSeparatingFunction
staticTokens: staticTokens
dynamicTokens: dynamicTokens
possibleDefaultTokens: possibleDefaultTokens
tokenHandlers: tokenHandlers

Examples

struct CalculatorLexer
{
	 mixin Lexer!(IdType, Token, defaultTokenFunction, isSeparating,
		 staticTokens, dynamicTokens, possibleDefaultTokens, tokenHandlers);

	 this (ubyte[] bytes)
	 {
		 this.range = LexerRange(bytes);
		 popFront();
	 }

	 void popFront() pure
	 {
		 _popFront();
	 }

	 Token lexNumber() pure nothrow @safe
	 {
		 // implementation goes here
	 }

	 Token lexWhitespace() pure nothrow @safe
	 {
		 // implementation goes here
	 }

	 Token defaultTokenFunction() pure nothrow @safe
	 {
		 // There is no default token in the example calculator language, so
		 // this is always an error.
		 range.popFront();
		 return Token(tok!"");
	 }

	 bool isSeparating(size_t offset) pure nothrow @safe
	 {
		 // For this example language, always return true.
		 return true;
	 }
}