Source files and tokenization

Tokens

A token is a series of contiguous characters that the compiler treats as a unit. Translation phase 3 partitions a source file into a sequence of tokens. Tokens fall into seven classes:

• Identifiers

• Keywords

• Numeric Constants

• Character Constants

• String literals

• Operators

• Other separators and punctuators