Example 2
Here, the two assignment statements are tokenized ,
that is, broken up into their elementary parts. Looking at the output, the reader may understand why the output is often called a token stream. In
reality, this token stream would not be printed out except as a debugging aid
when the compiler is in the process of being generated (debugged).
