Compiler/lexical analyzer: Difference between revisions

: ''Lexical analysis is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (strings with an identified "meaning"). A program that performs lexical analysis may be called a lexer, tokenizer, or scanner (though "scanner" is also used to refer to the first stage of a lexer).''

Create a lexical analyzer for the simple programming language specified below. The

if two versions of the solution are provided: One without the lexer module, and one with.

The simple programming language to be analyzed is more or less a subset of [[C]]. It supports the following tokens:

</pre>

The program output should be a sequence of lines, each consisting of the following whitespace-separated fields:

# the token value (only for <tt>IDENTIFIER</tt>, <tt>INTEGER</tt>, and <tt>STRING</tt> tokens)

The following error conditions should be caught:

|}

{| class="wikitable"

|}

The Flex, C, Python and Euphoria versions can be considered reference implementations.