Scanning

Scanning · Crafting Interpreters

This task has been variously called “scanning” and “lexing” (short for “lexical analysis”) over the years. Way back when computers were as big as Winnebagos but had less memory than your watch, some people used “scanner” only to refer to the piece of code that dealt with reading raw source code characters from disk and buffering them in memory. Then “lexing” was the subsequent phase that did useful stuff with the characters.

https://craftinginterpreters.com/scanning.html

Framework

First read the file into memory

If you want to run in an interactive mode where you type each line into a terminal, you can also do this, and it is called REPL pronounced like rep-el

Read each line at a time, and call your scanner class to scanTokens

  private static void run(String source) {
    Scanner scanner = new Scanner(source);
    List<Token> tokens = scanner.scanTokens();

    // For now, just print the tokens.
    for (Token token : tokens) {
      System.out.println(token);
    }
  }

Error handling

Its a good idea to create a error handling class / logging class.

It could include severity, the line number, where the error was and a message

Lexemes and Tokens

A lexeme is a substring that represents something

Next just by looking at a lexeme we want to determine what type of lexeme it is

Examples:

enum TokenType {
  // Single-character tokens.
  LEFT_PAREN, RIGHT_PAREN, LEFT_BRACE, RIGHT_BRACE,
  COMMA, DOT, MINUS, PLUS, SEMICOLON, SLASH, STAR,

  // One or two character tokens.
  BANG, BANG_EQUAL,
  EQUAL, EQUAL_EQUAL,
  GREATER, GREATER_EQUAL,
  LESS, LESS_EQUAL,

  // Literals.
  IDENTIFIER, STRING, NUMBER,

  // Keywords.
  AND, CLASS, ELSE, FALSE, FUN, FOR, IF, NIL, OR,
  PRINT, RETURN, SUPER, THIS, TRUE, VAR, WHILE,

  EOF
}

When getting Lexemes we also want to determine metadata

class Token {
  final TokenType type;
  final String lexeme;
  final Object literal;
  final int line;

  Token(TokenType type, String lexeme, Object literal, int line) {
    this.type = type;
    this.lexeme = lexeme;
    this.literal = literal;
    this.line = line;
  }

  public String toString() {
    return type + " " + lexeme + " " + literal;
  }
}

After we have assigned our Lexeme metadata we can now call it a token, because it is labled

Regular Languages and Expressions

To find Tokens we take in a stream of chars and slowly assemble them into entire Tokens, and then spit it out

Alternatively we can throw a bunch of regex to find each token

The Scanner Class

If we go with the non-regex method, then we might design our scanner class like so

1. Loop though the file char by char until end of file

2. Maintain a start (of current lexeme), current (what we char we are looking at), and line (the current line we are looking at) pointers

Recognising Lexemes

https://craftinginterpreters.com/scanning.html#recognizing-lexemes

I now recommend reading the book, there is not much point in condensing this but essentially the book recommends a switch case with look ahead for tokens like ! because it might be actually !=

Just note that at this stage, you want to throw errors you find early, such as unexpected char

Also remember that // may mean comment thus you skip the line, by consuming until you reach the end of the line. Technically speaking comments are lexemes but they are meaningless so we ignore them.

For strings its important to note that we need not only to get the token, but we also get the value to be later used by the interpreter. This means we need to consider \n ect.

For numbers you will need to consider decimals and -

Maximal Munch - is the idea that you should consume as many chars as possible before creating a token. Thus ---a should become the tokens -- - a