Programming, Compilers, Interpreters, Programming Languages.
language, for example, keywords such as if, while or func, identifiers, strings, numbers, operators or single characters like (, ), . or :. A token is an object describing a lexeme. Along with the value of the lexeme (the actual string of characters of the lexeme), it contains information such as