Lexical Analysis

Lexical Analysis
Lexical Analysis What do we want to do? Example:

if (i == j)
else
z = 0; z = 1;
The input is just a sequence of characters:

\tif (i == j)\n\t\tz = 0;\n\telse\n\t\tz = 1;
Goal: Partition input string into substrings

And classify them according to their role
Whats a Token? Output of lexical analysis is a stream of tokens A token is a syntactic category
In English:
noun, verb, adjective,
In a programming language:
Identifier, Integer, Keyword, Whitespace,
Parser relies on the token distinctions:

E.g., identifiers are treated differently than keywords
Tokens Tokens correspond to sets of strings:

Identifiers: strings of letters or digits, starting
with a letter Integers: non-empty strings of digits Keywords: else or if or begin or Whitespace: non-empty sequences of blanks, newlines, and tabs OpenPars: left-parentheses
Lexical Analyzer: Implementation An implementation must do two things:

1. Recognize substrings corresponding to tokens
2. Return:
1. The type or syntactic category of the token, 2. the value or lexeme of the token (the substring itself).
Example
Our example again:
\tif (i == j)\n\t\tz = 0;\n\telse\n\t\tz = 1;
Token-lexeme pairs returned by the lexer:

(Whitespace, \t) (Keyword, if) (OpenPar, () (Identifier, i) (Relation, ==) (Identifier, j)
Lexical Analyzer: Implementation The lexer usually discards uninteresting tokens that dont contribute to parsing. Examples: Whitespace, Comments
Lookahead. Two important points:

1. The goal is to partition the string. This is implemented by reading left-to-right, recognizing one token at a time
2. Lookahead may be required to decide where one token ends and the next token begins Even our simple example has lookahead issues i vs. if = vs. ==
Next We need
A way to describe the lexemes of each token
A way to resolve ambiguities
Is if two variables i and f? Is == two equal signs = =?
Regular Languages There are several formalisms for specifying tokens Regular languages are the most popular
Simple and useful theory Easy to understand Efficient implementations
Languages
Def. Let S be a set of characters. A language over S is a set of strings of characters drawn from S (S is called the alphabet )
Examples of Languages
Alphabet = English characters Language = English sentences Not every string on English characters is an English sentence Alphabet = ASCII Language = C programs
Note: ASCII character set is different from English character set
Notation Languages are sets of strings. Need some notation for specifying which sets we want For lexical analysis we care about regular languages, which can be described using
regular expressions.
Regular Expressions and Regular Languages Each regular expression is a notation for a regular language (a set of words) If A is a regular expression then we write L(A) to refer to the language denoted by A
Regular Expressions Single character: c L(c) = { c } (for any c S) Concatenation: AB (where A and B are reg. exp.) L(AB) = { ab | a L(A) and b L(B) }
Compound Regular Expressions Union

L(A | B) = L(A) L(B)
= { s | s L(A) or s L(B) } Examples:

if | then | else = { if, then, else} 0 | 1 | | 9 = { 0, 1, , 9 }
Another example:
L((0 | 1) (0 | 1)) = { 00, 01, 10, 11 }
More Compound Regular Expressions So far we do not have a notation for infinite languages Iteration: A* L(A*) = { } | L(A) | L(AA) | L(AAA) | Examples:
0* = { , 0, 00, 000, } 1 0* = { strings starting with 1 and followed by 0s }
Epsilon:
L() = { }
Example: Keyword Keyword: else or if or begin or
else | if | begin |
Example: Integers Integer: a non-empty string of digits digit = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 number = digit digit*
Example: Identifier Identifier: strings of letters or digits,
starting with a letter
letter = A | | Z | a | | z identifier = letter (letter | digit) * Is (letter* | digit*) the same as (letter | digit) * ?
Example: Whitespace Whitespace: a non-empty sequence of blanks,
newlines, and tabs
( | \t | \n)+
Finite Automata Specifying lexical structure using regular expressions create finite automata for recognizing RE Finite automata
Deterministic Finite Automata (DFAs) Non-deterministic Finite Automata (NFAs)
Implementation of regular expressions

RE => NFA => DFA => Tables
Regular Expressions 1. Select a set of tokens

Number, Keyword, Identifier, ...
2. Write a R.E. for the lexemes of each token

Number = digit+ Keyword = if | else | Identifier = letter (letter | digit)* OpenPar = (
Regular Expressions 3. Construct R, matching all lexemes for all tokens R = Keyword | Identifier | Number | = R1 | R2 | R3 | Facts: If s L(R) then s is a lexeme
Furthermore s L(Ri) for some i This i determines the token that is reported
Regular Expressions 4. Let the input be x1xn
(x1 ... xn are characters in the language alphabet) For 1 i n check x1xi L(R) ?
x1xi L(Rj) for some i and j
5. It must be that
6. Remove x1xi from input and go to (4)
Lexing Example R = Whitespace | Integer | Identifier | + Scanning f+3 +g

f matches R, more precisely Identifier + matches R, more precisely + The token-lexeme pairs are
(Identifier, f), (+, +), (Integer, 3) (Whitespace, ), (+, +), (Identifier, g)
We would like to drop the Whitespace tokens

after matching Whitespace, continue matching
Finite Automata Regular expressions = specification Finite automata = implementation A finite automaton consists of
An input alphabet S A set of states S A start state n A set of accepting states F S A set of transitions state input state
Finite Automata Transition
Is read In state s1 on input a go to state s2 If end of input

If in accepting state => accept, otherwise => reject
a s1 s2
If no transition possible => reject
Finite Automata State Graphs A state The start state An accepting state A transition
a
A Simple Example A finite automaton that accepts only 1

1
A finite automaton accepts a string if we can follow transitions labeled with the characters in the string from the start to some accepting state
Another Simple Example A finite automaton accepting any number of 1s followed by a single 0 Alphabet: {0,1}
1 0
Check that 1110 is accepted but 110 is not
Epsilon Moves Another kind of transition: -moves

A B
Machine can move from state A to state B without reading input
Deterministic and Nondeterministic Automata Deterministic Finite Automata (DFA)

One transition per input per state No -moves
Nondeterministic Finite Automata (NFA)

Can have multiple transitions for one input in a given state Can have -moves
Execution of Finite Automata A DFA can take only one path through the state graph
Completely determined by input
NFAs can choose

Whether to make -moves Which of multiple transitions for a single input to take
NFA vs. DFA (1) NFAs and DFAs recognize the same set of languages (regular languages)
DFAs are easier to implement

There are no choices to consider
NFA vs. DFA (2) For a given language the NFA can be simpler than the DFA
1
NFA
1 0
DFA
0 1
DFA can be exponentially larger than NFA
Regular Expressions to Finite Automata
NFA Regular expressions
DFA
Lexical Specification
Table-driven Implementation of DFA
Regular Expressions to NFA (Thompson's Algorithm )

For each kind of re, define an NFA
Notation: NFA for re A A
For
For input a
Regular Expressions to NFA (2) For AB

A
For A | B
Regular Expressions to NFA (3) For A*
Example of RE -> NFA conversion Consider the regular expression (1 | 0)*1 The NFA is
A C
0 F D
G H I
Example of RE -> NFA conversion
Deterministic Finite Automata (DFA)

A deterministic finite automaton (DFA) is a special case of an NFA where There are no moves on input . For each state s and input symbol a, there is one edge out of a labeled a.
Converting the NFA into a DFA
Transition Table
Design of a lexical analyzer generator(lex)
Lex Specifications
A Lex program consists of three parts: 1. Declarations %% 2. Translation Rules %% 3. Auxiliary procedures

Lexical Analysis

Uploaded by

Copyright:

Available Formats

Lexical Analysis

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lexical Analysis

Uploaded by

Copyright:

Available Formats

Lexical Analysis

Lexical Analysis What do we want to do? Example:

The input is just a sequence of characters:

Goal: Partition input string into substrings

Parser relies on the token distinctions:

Tokens Tokens correspond to sets of strings:

Identifiers: strings of letters or digits, starting

Lexical Analyzer: Implementation An implementation must do two things:

Token-lexeme pairs returned by the lexer:

Lookahead. Two important points:

Note: ASCII character set is different from English character set

Compound Regular Expressions Union

= { s | s L(A) or s L(B) } Examples:

Example: Keyword Keyword: else or if or begin or

Example: Identifier Identifier: strings of letters or digits,

starting with a letter

Example: Whitespace Whitespace: a non-empty sequence of blanks,

newlines, and tabs

Implementation of regular expressions

Regular Expressions 1. Select a set of tokens

2. Write a R.E. for the lexemes of each token

Regular Expressions 4. Let the input be x1xn

Lexing Example R = Whitespace | Integer | Identifier | + Scanning f+3 +g

We would like to drop the Whitespace tokens

Finite Automata Transition

Is read In state s1 on input a go to state s2 If end of input

If no transition possible => reject

A Simple Example A finite automaton that accepts only 1

Check that 1110 is accepted but 110 is not

Epsilon Moves Another kind of transition: -moves

Machine can move from state A to state B without reading input

Deterministic and Nondeterministic Automata Deterministic Finite Automata (DFA)

Nondeterministic Finite Automata (NFA)

NFAs can choose

DFAs are easier to implement

DFA can be exponentially larger than NFA

Regular Expressions to Finite Automata

NFA Regular expressions

Table-driven Implementation of DFA

Regular Expressions to NFA (Thompson's Algorithm )

Regular Expressions to NFA (2) For AB

Regular Expressions to NFA (3) For A*

Example of RE -> NFA conversion

Deterministic Finite Automata (DFA)

Converting the NFA into a DFA

Design of a lexical analyzer generator(lex)

You might also like