| 
 
 
 
 
 
 
 
 
 
  
 
 
 
   | 
    | 
 
 
 
     
 
 
  Example 2  
   
  
 
 
 
 Here, the two assignment statements are tokenized ,
 that is, broken up into their elementary parts. Looking at the output, the reader may understand why the output is often called a token stream.  In
 reality, this token stream would not be printed out except as a debugging aid
 when the compiler is in the process of being generated (debugged).
 
 
 
 
 
 
 
 
 
 
 
 
   
 
 
   |