Compiler/lexical analyzer: Difference between revisions

Content added Content deleted
Line 4,886: Line 4,886:
</lang>
</lang>


This script uses <i>lexer.tokenize_text</i> to show the tokens produced a source text.
This script uses <i>lexer.tokenize_text</i> to show the token sequence produced from a source text.


<lang Lua>lexer = require 'lexer'
<lang Lua>lexer = require 'lexer'
Retrieved from "https://rosettacode.org/wiki/Compiler/lexical_analyzer"