Lexical Analyzer

Thursday, November 20th, 2008

Modules and Algorithms:

Functions:
Verify(): The lexical analyzer uses the verify operation to determine whether there is entry for a lexeme in the symbol table.

Data structures:

Structure : A structure in C is a collection of variables which contains related data items of similar and /or dissimilar data types but logically related items. Each variable in the structure represents an item and is called as a member or field of the structure. Complex data can be important represented in more meaningful way using structures and is one of the very features available in C.

An array of structures is basically a array of records. Each an every record has the same format consisting of similar or dissimilar data types but logically related entities.

Algorithms:

Procedure main
begin
if symb=’#’ then
begin
advance to next token in input file
if symb=’i’ then
begin
advance to next token in input file
while symb!=’n’ do
begin
advance to next token in input file
end {while }
print symb is a preprocessor directive
end {if symb=’i’}
if symb=’d’ then
begin
advance to next token input file
while symb!=’ ‘ do
begin
advance to next token in input file
end{while}
advance to next token in input file
print symb is a constant
advance to next token in input file
while symb!=’n’ do
begin
advance to the next token in input file
end {while}
end {if symb=’d’}
end {if symb=’#’}
if symb is a alphabet or symb=’_’ then
begin
advance to the next token in input file
while symb is a digit or alphabet or symb=’_’ do
begin
advance to the next token of input file
end {while}
call function verify to check whether symb is a identifier or keyword
end {if}
if symb=’+’ then
begin
advance to the next token in input file
if symb=’+’
print symb is ++ operator
else
ungetc symb from the input file
print symb is + operator
end {if}
if symb=’-’ then
begin
advance to the next token in input file
if symb=’-’
print symb is — operator
else
ungetc symb from the input file
print symb is – operator
end {if}
if symb=’|’ then
begin
advance to the next token in input file
if symb=’|’
print symb is logical or operator
else
ungetc symb from the input file
print symb is bitwise or operator
end {if}
if symb=’*’ then
begin
print symb is a multiplication operator
end {if}
if symb=’?’ then
begin
print symb is a conditional operator
end{if}
if symb=’!’or symb=’>’or symb=’<’then
begin
advance to the next token in input file
if symb=’=’
print symb is a relational operator
else
ungetc symb from output file
print symb is a operator
end{if}
if symb=’=’
begin
advance to next token in input file
if symb=’=’then
print symb is equal to operator
else
ungetc symb from output file
print symb is assignment operator
end{if}
if symb=’&’ then
begin
advance to next token in input file
if symb=’&’ then
print symb is a logical and operator
else
print & symb is an address operator
end{if}
if symb=’/’ then
begin
advance to next token in input file
if symb=’*’ then
begin
advance to next token in input file
while symb!=’/’ do
advance to next token in input file
end{while}
end{if}
else if symb=’/’ then
begin
advance to next token in input file
while symb!=’n’ do
advance to next token in input file
end{while}
end{if}
else
ungetc symb from output file
print symb is a division operator
end{if}
if symb is a digit then
begin
advance to next token in input file
while symb is a digit or symb=’.’ then
begin
advance to next token in input file
end {while}
print symb is a number
end{if}
if symb =’”’ then
begin
advance to next token in input file
while symb!=’”’ do
begin
advance to next token in input file
end{while}
print symb is a string
end{if}}
if symb= ‘{‘ then
print open brace
if symb=’}’ then
print close brace
if symb=’[‘ then
print open bracket
if symb=’]’ then
print close bracket
if symb=’(‘ then
print open parenthesis
if symb=’)’ then
print close parenthesis
end {procedure main}
procedure verify
begin
scan the symbol table to check if encountered token exists
if exists
return token value
end{procedure}

Pages: 1 2 3

Author Name :
Ranjith

Total : 19 Comments

Description :

Browse other articles of this author

Simillar Posts

19 Responses to Lexical Analyzer

Sanjay Tiwari
February 20th, 2009 at 9:40 am
I want system programming, program code in C/C++

Thank you
tiwari
Samana
March 8th, 2009 at 9:21 pm
Hello ..plz give me the code in C…I want it urgent..plz send it at sonakhan20032003@yahoo.com
Thanks a lot
priyank kumar
September 1st, 2009 at 11:52 pm
Hello ..plz give me the code in turbo C for the sudoko solver…I want it urgent..plz send it at priyank07310@yahoo.com
Thanks a lot
yus
September 28th, 2009 at 9:24 am
hello… plz help me the code in C++ and get the token in lexical analysis
ashmy
October 1st, 2009 at 10:52 pm
plz give me the code in lex to recognize the tokens
madhukiran R
October 24th, 2009 at 8:57 pm
please give source code of lexical analyzer using c
Mithil
November 5th, 2009 at 2:54 am
Hey can plz mail me the lex code to identify different tokens from a c code..
thanx in advance..
Arati Patro
December 9th, 2009 at 2:58 pm
\\ Token Separation \\

Write a program to identify and generate the tokens present in the given input
#include
#include
#include
#include

int key = 0;
char expr[100];
char cont[][20]={“CONTROLS”,”for”,”do”,”while”,”NULL”,};
char cond[][20]={“CONDITION”,”if”,”then”,”NULL”};
char oprt[][20]={“OPERATOR”,”+”,”-”,”*”,”/”,”%”,”<","”,”>=”,”=”,”(“,”)”,”NULL”};
char branch[][20]={“BRANCHING”,”goto”,”jump” ,”NULL”};
void checking(char[],char[][20]);
void main()
{
int i,j,l,k,m,n;
char sbexpr[50],txt[3];
clrscr();
cout<=97 && sbexpr[m]=65 && sbexpr[m]<=90)))
{
cout<<"\n"<<sbexpr[m]<”<<"Identifier\n";
key = 1;
}
}
}
if(key == 0)
{
cout<<"\n"<<sbexpr<”<<"Address\n";
key = 1;
}
}

getch();
}

void checking (char expr[],char check[][20])
{
for(int i=1;strcmp(check[i],"NULL")!=0;i++)
{
if(strcmp(expr,check[i])==0)
{
cout<<expr<”<<check[0]<<"\n";
key = 1;
}
}
}
Anonymous
December 10th, 2009 at 3:16 pm
#include
#include
#include
#include

void Open_File();
void Demage_Lexeme();
int Search(char[256],int);
void analyze();
void Skip_Comment();
void Read_String();
void Is_Keyword_Or_Not();
void Is_Identifier_Or_Not();
void Is_Operator_Or_Not();
void Read_Number();
void Is_Special_Or_Not();
void Is_Comparison_Or_Not();
void Add_To_Lexical (char[256],int,char[256]);
void Print_ST();
void Print_TOKEN();
void Token_Attribute();
truct lexical
{
char data[256]; //Value of token.
int line[256]; //Line # which token appear in input
file.
int times; //# of times that token appear in input
file.
char type[256]; //Type of each token.
struct lexical *next;
};

typedef struct lexical Lex;
typedef Lex *lex;

/****************************************************************
File pointer for accessing the file.
*****************************************************************/

FILE *fp;
FILE *st;
FILE *token;
char lexeme[256],ch;
int f,flag,line=1,i=1;
lex head=NULL,tail=NULL;

/****************************************************************
Array holding all keywords for checking.
*****************************************************************/

char
*keywords[]={“procedure”,”is”,”begin”,”end”,”var”,”cin”,”cout”,”if”,
“then”,”else”,”and”,”or”,”not”,”loop”,”exit”,”when”,
“while”,”until”};

/****************************************************************
Array holding all arithmetic operations for checking.
*****************************************************************/

char arithmetic_operator[]={‘+’,'-’,'*’,'/’};

/****************************************************************
Array holding all comparison operations for checking.
*****************************************************************/

char *comparison_operator[]={“”,”=”,”<=","”,”>=”};

/****************************************************************
Array holding all special for checking.
*****************************************************************/

char special[]={‘%’,'!’,'@’,'~’,'$’};

/****************************************************************

**************
*MAIN PROGRAM*
**************

*****************************************************************/

void main()
{
Open_File();
analyze();
fclose(fp);
Print_ST();
Print_TOKEN();
}

/****************************************************************
This function open input sourse file.
*****************************************************************/

void Open_File()
{

fp=fopen(“source.txt”,”r”); //provide path for source.txt here
if(fp==NULL)
{
printf(“!!!Can’t open input file – source.txt!!!”);
getch();
exit(0);
}
}

/****************************************************************
Function to add item to structure of array to store data and
information of lexical items.
*****************************************************************/

void Add_To_Lexical (char value[256],int line,char type[256])
{
lex new_lex;

if (!Search(value,line)) //When return 1 the token not found.
{

new_lex=malloc(sizeof(Lex));

if (new_lex!=NULL)
{
strcpy(new_lex->data,value);
new_lex->line[0]=line;
new_lex->times=1;
strcpy(new_lex->type,type);
new_lex->next=NULL;

if (head==NULL)
head=new_lex;
else
tail->next=new_lex;

tail=new_lex;
}
}
}

/****************************************************************
Function to search token.
*****************************************************************/

int Search (char value[256],int line)
{
lex x=head;
int flag=0;

while (x->next!=NULL && !flag)
{
if (strcmp(x->data,value)==0)
{
x->line[x->times]=line;
x->times++;
flag=1;
}
x=x->next;
}
return flag;
}

/****************************************************************
Function to print the ST.TXT .
*****************************************************************/

void Print_ST()
{
lex x=head;
int j;

if ((st=fopen(“ST.TXT”,”w”))==NULL)
printf(“The file ST.TXT cat not open.
“);

else

{
fprintf(st,” %s %s %s
“,”Line#”,”Lexeme”,”Type”);
fprintf(st,” —- —— —-
“);

while (x!=NULL)
{
if ((strcmp(x->type,”num”)==0) ||
(strcmp(x->type,”keyword”)==0) ||
(strcmp(x->type,”identifier”)==0))
{
fprintf(st,” “);

for (j=0;jtimes;j++)
{
fprintf(st,”%d”,x->line[j]);
if (j!=x->times-1) //This condition to prevent the comma
fprintf(st,”,”,x->line[j]); //”,” to not print after last line #.
}

fprintf(st,” %-6s %-6s
“,x->data,x->type);
}
x=x->next;
}

fclose(st);
}
}

/****************************************************************
Function to print the TOKENS.TXT .
*****************************************************************/

void Print_TOKEN()
{
int flag=0;

fp=fopen(“source.txt”,”r”);

if(fp==NULL)
{
printf(“!!!Can’t open input file – source.txt!!!”);
getch();
exit(0);
}

else

{
if ((token=fopen(“TOKENS.TXT”,”w”))==NULL)
printf(“The file ST.TXT cat not open.
“);

else

{
ch=fgetc(fp);

while (!(feof(fp)))
{

if (ch==’ ‘ && !flag)
{
do
ch=fgetc(fp);
while (ch==’ ‘);

fseek(fp,-2,1);
ch=fgetc(fp);
flag=1;
}

if (ch!=’
‘ && ch!=’ ‘)
fprintf(token,”%c”,ch);

if (ch==’
‘)
{
fprintf(token,”
“);
Token_Attribute();
i++;
flag=0;
}

ch=fgetc(fp);
}
}
}
fclose(fp);
fclose(token);
}

/****************************************************************
Function to put the token and atrribute in TOKENS.TXT .
*****************************************************************/

void Token_Attribute()
{
lex x=head;
int j;

while (x!=NULL)
{
if (x->line[0]==i)
{
fprintf(token,”token : %-4s “,x->type);

if ((strcmp(x->type,”num”)==0) ||
(strcmp(x->type,”keyword”)==0) ||
(strcmp(x->type,”identifier”)==0))

{
fprintf(token,”attribute : line#=%-4d
“,i);
}

else

{
fprintf(token,”attribute : %-4s
“,x->data);
}

}
x=x->next;
}
fprintf(token,”
“);
}

/****************************************************************
Function to create lexical analysis.
*****************************************************************/

void analyze()
{

ch=fgetc(fp); //Read character.

while(!feof(fp)) //While the file is not end.
{

if(ch==’
‘) //Compute # of lines in source.txt
.
{
line++;
ch=fgetc(fp);
}

if(isspace(ch) && ch==’
‘ )
{
line++;
ch=fgetc(fp);
}
if(isspace(ch) && ch!=’
‘ ) //The character is space.
ch=fgetc(fp);

if(ch==’/’ || ch==’”‘) //Function for skipping comments in the
file
Skip_Comment(); //and ‘”‘ with display statements.

if(isalpha(ch)) //The character is leter.
{
Read_String();
Is_Keyword_Or_Not();
Is_Operator_Or_Not();
Is_Identifier_Or_Not();
}

if(isdigit(ch)) //The character is digit.
Read_Number();

if (ch==’;') //The character is semicolon.
Add_To_Lexical(“;”,line,”semicolon”);

if (ch==’:') //The character is colon.
Add_To_Lexical(“:”,line,”colon”);

if (ch==’,') //The character is comma.
Add_To_Lexical(“,”,line,”comma”);

if (ch==’(‘) //The character is parenthesis.
Add_To_Lexical(“(“,line,”parenthesis”);

if (ch==’)') //The character is parenthesis.
Add_To_Lexical(“)”,line,”parenthesis”);

//The character is comparison_operator
if (ch==”)
Is_Comparison_Or_Not();

Is_Special_Or_Not(); //After failed scaning in before cases
//check the character is special or not.
Demage_Lexeme();

if(isspace(ch) && ch==’
‘ )
{
line++;
ch=fgetc(fp);
}
else
ch=fgetc(fp);
}
}

/****************************************************************
This function read all character of strings.
*****************************************************************/

void Read_String()
{
int j=0;

do
{
lexeme[j++]=ch;
ch=fgetc(fp);
} while(isalpha(ch));

fseek(fp,-1,1);
lexeme[j]=’
Roshini
December 12th, 2009 at 2:29 pm
program 4 token separation in c++
Roshini
December 12th, 2009 at 2:33 pm
Token is a sequence of characters that has a collective meaning
oil field trash
February 2nd, 2010 at 11:39 pm
I found this kit on ebay =) No problem =)
gavincurtis
February 3rd, 2010 at 6:35 am
“speak Japanese
i cant understand at all”
Ankit
March 3rd, 2010 at 7:17 pm
Can you tell how many tokes the following statments consists of:-

printf(“Test of %d tokens %d”,a,b);

printf(“TEST”);

??

i am bit confused weather %d will be treated as separate tokens or not?

And for the literal weather the quotations (“) will be counted as tokens or not?
ramya
April 4th, 2010 at 6:48 pm
plz send me the source code in lex to identify different parts of speech by making use of symbol table.
Magoba
January 23rd, 2011 at 1:58 am
Let the minds continue to convene here and devour problems!!

Thanx to you all enthusiasts!
pushpanjali.c.h.
May 11th, 2011 at 7:05 am
plz send me the source code in lex to identify name of city and persons
disnad
July 10th, 2011 at 10:51 pm
please send me a program in c++ that give a cpp file and separate token and type of each and create a symbol table with error report field. (lexical analyzer)

my email: sam_disnad@yahoo.com

Thanx to you
kapil
October 8th, 2011 at 3:42 pm
on 10oct is my exam please send me notes compiler desin

Projects

Basic Electronics Computer Programming Microcontrollers

Question and Answer

C/C++ Unix & Linux Wordpress

Source codes

C C++ Java

Articles

PHP PCI protocol Microcontrollers Analog

eBooks

C Boks C++ Books Java Books SQL Books

Free email signup

Remote System Monitoring and Controlling via Web based Mobile or Desktop Application

Dependency Injection in C# .NET

what is Void Pointer?

What is dangling pointer?

Delegate and Events in C# .NET

Mobile Information Device Profile (MIDP 2.0) in J2ME

Best Practices for Small computing devices

Java APIs for Bluetooth Wireless Technology (JABWT)

Midlet in J2ME

Record management in J2ME MIDP

8951 8951 Microcontroller algorithm Algorithms array binary tree C C/C++ C/C++ Programms c graphics c interview question class cpp c program c programs Data structure download engineering questions free frequently asked interview questions Graphics initgraph Insertion sort interview questions and answers it interview questions Java Java program Keyboard Interface lab programs linked list Linked Lists Microcontroller non-recursive postfix program programs questions for an interview Queue source code Source Codes stack Stepper motor struct templates top interview questions

Recent Members

This is the one stop educational site for all Electronic and Computer students. If you want to learn something new then we are here to help. We work on Microcontroller projects, Basic Electronics, Digital electronics, Computer projects and also in basic c/c++ programs.

#Home #Sitemap #Submit #Terms of Use