3.5 Generating a Lexical Analyzer vs.
Writing a Lexical Analyzer

The reader may think it is much harder to write a lexical analyzer generator than it is just to write a lexical analyzer and then make changes to it to produce a different lexical analyzer. After all, most programming languages have similar tokens. This thought has been voiced by many compiler experts. In fact, many compiler tools allow the user to write lexical analyzer and call it from the generated parser or to make changes to a lexical analyzer provided by the tool.

Nevertheless, the process is informative, and there may be applications for which the user may wish to be able to generate various lexical analyzers.

The next section shows an example of a lexical analyzer generator, LEX (Lesk and Schmidt, 1975), a tool which comes with UNIX.