AMBO UNIVERSITY, IINSSTITUTE of TECHNOLOGY

SCHOOL OF INFORMATICS and ELECTRICAL ENGINEERING

Instructor: Fikru Tafesse (MSc.)

1
 A Typical Lexical Analyzer Generator
• For efficient design of compiler, various tools have been built for constructing lexical
analyzers using special purpose notations called regular expressions.

• Regular expressions are used in recognizing tokens.

• A tool called LEX gives this specification.

• LEX is utility which generates the lexical analyzer.

• In LEX tool, designing the regular expressions for corresponding tokens is a logical
task.

• Having a file with extension. l (pronounced as dot l) the lex specifications file can be
created.
2
 Cont'd …
• For Example: the specification file name firstLab.l.
• This firstLab.l is the given to LEX compilers to produce lex.yy.c
• This lex.yy.c is a C++ program which is actually a lexical analyzer program.
• The specification file stores the regular expressions for the tokens, the lex.yy.c file
consists of the tabular representation of the transition diagrams constructed for the
regular expressions of specification file say firstLab.l.
• The lexemes can be recognized with the help of tabular transitions diagrams and
some standards routines.
• In the specification file of LEX actions are associated with each regular expression.
• This action are simply pieces of C++ codes.
3
 Cont'd …
• These pieces of C++ code are directly carried over to the lex.yy.c.
• Finally, the C++ compiler compiles this generated lex.yy.c and produces an object program a.exe.
• When some input stream is given to a.exe then sequence of tokens gets generated.
• The above discussion scenario is modeled as below:

4
 Cont'd …
Lex Program
• Lex is a program that generates lexical analyzer.
• It is used with YACC parser generator.
• The lexical analyzer is a program that transforms an input stream into a sequence of tokens.
• It reads the input stream and produces the source code as output through implementing the
lexical analyzer in the C++ program.
Function of Lex
• Firstly, lexical analyzer creates a program firstLab.l in the Lex language.
• Then Lex compiler runs the firstLab.l program and produces a C++ program lex.yy.c.
• Finally, C++ compiler runs the lex.yy.c program and produces an object program a.exe. a.exe is
lexical analyzer that transforms an input stream into a sequence of tokens.
5
 Cont'd …
Lex File Format
• A Lex program is separated into three sections by %% delimiters.
• The format of Lex source is as follows:
%{
Declaration Section
%}
%%
Rule Section
%%
Auxiliary Procedure Section
6
 Cont'd …
Declaration section
• declaration of variables constants can be done.
• Some regular definitions can also be written in this section.
• The regular definitions are basically components of regular expressions appearing in the rule
section.
Rule Section
• The rule section consists of regular expressions with associated actions.
• The translation rules can be given in the form as:
R1 {action1}
R1 {action2}
. . .
Rn {actionn}
7
 Cont'd …
Auxiliary procedure section
• Auxiliary procedure section in which all the required procedure is defined.
• Sometimes these procedures are required by the actions in the rule section.
• The lexical analyzer or scanner works in co-ordination with parser.
• When activates by the parser, the lexical analyzer begins reading its remaining input,
character by character at a time.
• When the string is matched with one of the regular expressions Ri then corresponding
actioni will get executed and this actioni returns the control to the parser.
• To repeated search for the lexeme can be made in order to return all the tokens in the
source string.
8
• The lexical analyzer ignores white spaces and comments in this process.
 Cont'd …
Sample Lex Program to understand the above concept

9
 Cont'd …
• This a simple program that displays the given string "Welcome to Compiler Design Lab
Session “).
• In main function call to yylex routine is given.
• This function is defined in lex.yy.c program.
• yytext: when lexer matches or recognizes the token from input token then the lexeme is
stored in a null terminated string called yytext.
• Is called when scanner encounters end of file.
• yylex (): is important function.
• As soon as call to yytext () is encountered scanner starts scanning the source program.
• yywrap(): is called when scanner encounters end of file. if yywrap() returns 0 then scanner
continues scanning. 10
 Cont'd …
• When yywrap() returns 1 that means end of file is encountered.
• yyin: it is standard input file that stores input source program.
• yyleng: when a lexer recognizes token then the lexeme is stored in a null terminated string
called yytext.
• And yyleng stores the length or number of characters in the input string.
• The value in yyleng is same as strlen() functions.

• Before going to run and execute the above lex program you have to install
and configure some softwares like codeblocks, flex and others as follows in
my case.

11
 Cont'd …
1. First install codeblocks in appropriate directory.
2. Install flex in appropriate directory.
3. Set paths as the following:
4. Goto CodeBlocks->MinGW->bin copy the address of bin it somewhat look like
C:\Program Files (x86) \CodeBlocks\MinGW\bin
• Open control panel->goto System->Advanced Settings->Environment
Variables->System Variables click on path which inside system variables-
>click on edit->click on new and paste the copied path to it C:\Program Files
(x86)\CodeBlocks\MinGW\bin
• Press OK
12
 Cont'd …

5. Goto GnuWin32->bin copy the address of bin it should somewhat look like
this C:\GnuWin32\bin
6. Open control panel->goto System->Advanced Settings-> Environment
Variables->System Variables click on path which inside system variables-
>click on edit->click on new and paste the copied path to it C:\GnuWin32\
bin
• Now Let Write and Lex program and execute it
1. Create folder on Desktop with name LexProgram or any name you want.
2. Open notepad and type a Lex program
3. Save it inside the folder like filename.l 13
 Cont'd …

4. Make sure while saving save it as all files rather than as a text
document
5. Go to command prompt(cmd)
6. Go to the directory where you have saved the program
7. Type in command: flex filename.l
8. Type in command: g++ lex.yy.c
9. Execute/Run for windows command prompt: a.exe
Output look like below after executing the lex program on the next
Slide….
14
Cont'd …

15