THEORY OF COMPUTATION

What is TOC?
In theoretical computer science, the theory of computation is the branch that deals
with whether and how efficiently problems can be solved on a model of computation,
using an algorithm. The field is divided into three major branches: automata theory,
computability theory and computational complexity theory. In order to perform a
rigorous study of computation, computer scientists work with a mathematical
abstraction of computers called a model of computation. There are several models
in use, but the most commonly examined is the Turing machine.
Automata theory:
In theoretical computer science, automata theory is the study of abstract machines
(or more appropriately, abstract 'mathematical' machines or systems) and the
computational problems that can be solved using these machines. These abstract
machines are called automata. This automaton consists of
states (represented in the figure by circles),
and transitions (represented by arrows).
As the automaton sees a symbol of input, it makes a transition (or jump) to another
state, according to its transition function (which takes the current state and the
recent symbol as its inputs).
Uses of Automata: compiler design and parsing
Alphabets, Strings and Languages

Languages :
A general definition of language must cover a variety of distinct categories: natural languages,
programming languages, mathematical languages, etc. The notion of natural languages like
English, Hindi, etc. is familiar to us. Informally, language can be defined as a system suitable for
expression of certain ideas, facts, or concepts, which includes a set of symbols and rules to
manipulate these. The languages we consider for our discussion is an abstraction of natural
languages. That is, our focus here is on formal languages that need precise and formal
definitions. Programming languages belong to this category. We start with some basic concepts
and definitions required in this regard.

Symbols :
Symbols are indivisible objects or entity that cannot be defined. That is, symbols are the atoms of
the world of languages. A symbol is any single object such as , a, 0, 1, #, begin, or do.
Usually, characters from a typical keyboard are only used as symbols.

Alphabets :
An alphabet is a finite, nonempty set of symbols. The alphabet of a language is normally denoted
by . When more than one alphabets are considered for discussion, then subscripts may be
used (e.g.

etc) or sometimes other symbol like G may also be introduced.

Example :
Strings or Words over Alphabet :
A string or word over an alphabet

is a finite sequence of concatenated symbols of

Example : 0110, 11, 001 are three strings over the binary alphabet { 0, 1 } .

aab, abcb, b, cc are four strings over the alphabet { a, b, c }.

It is not the case that a string over some alphabet should contain all the symbols from the
alphabet. For example, the string cc over the alphabet { a, b, c } does not contain the symbols a
and b. Hence, it is true that a string over an alphabet is also a string over any superset of that
alphabet.

Length of a string :
The number of symbols in a string w is called its length, denoted by |w|.

Example : | 011 | = 4, |11| = 2, | b | = 1

It is convenient to introduce a notation e for the empty string, which contains no symbols at all.
The length of the empty string e is zero, i.e., | e | = 0.
Convention : We will use small case letters towards the beginning of the English alphabet to
denote symbols of an alphabet and small case letters towards the end to denote strings over an
alphabet. That is,

(symbols) and

are strings.

Some String Operations :

Let

and

be two strings. The concatenation

of x and y denoted by xy, is the string

. That is, the concatenation
of x and y denoted by xy is the string that has a copy of x followed by a copy of y without any
intervening space between them.
Example : Concatenation of the strings 0110 and 11 is 011011 and concatenation of the strings
good and boy is goodboy.
Note that for any string w, we = ew = w. It is also obvious that if | x | = n and | y | = m,
then | x + y | = n + m.

u is a prefix of v if v = ux for some string x.

u is a suffix of v if v = xu for some string x.
u is a substring of v if v = xuy for some strings x and y.

Example : Consider the string 011 over the binary alphabet. All the prefixes, suffixes and
substrings of this string are listed below.
Prefixes: e, 0, 01, 011.
Suffixes: e, 1, 11, 011.
Substrings: e, 0, 1, 01, 11, 011.
Note that x is a prefix (suffix or substring) to x, for any string x and e is a prefix (suffix or
substring) to any string.
A string x is a proper prefix (suffix) of string y if x is a prefix (suffix) of y and x y.
In the above example, all prefixes except 011 are proper prefixes.

Powers of Strings : For any string x and integer

, we use

to denote the string formed

by sequentially concatenating n copies of x. We can also give an inductive definition of

follows:

as

= e, if n = 0 ; otherwise

Example : If x = 011, then

= 011011011,

= 011 and

Powers of Alphabets :
We write
(for some integer k) to denote the set of strings of length k with symbols from
other words,
= { w | w is a string over

and | w | = k}. Hence, for any alphabet,

strings of length zero. That is,

. In

denotes the set of all

= { e }. For the binary alphabet { 0, 1 } we have the following.

he set of all strings over an alphabet

is denoted by

. That is,

The set
contains all the strings that can be generated by iteratively concatenating symbols
from
any number of times.

Example : If

= { a, b }, then

= { e, a, b, aa, ab, ba, bb, aaa, aab, aba, abb, baa, }.

Please note that if

, then
that is
. It may look odd that one can proceed from
the empty set to a non-empty set by iterated concatenation. But there is a reason for this and we
accept this convention.

The set of all nonempty strings over an alphabet

Note that

is denoted by

. That is,

is infinite. It contains no infinite strings but strings of arbitrary lengths.

Reversal :
For any string

the reversal of the string is

An inductive definition of reversal can be given as follows:

Languages :
A language over an alphabet is a set of strings over that alphabet. Therefore, a language L is
any subset of

. That is, any

is a language

Example :

1. F is the empty language.

2.
is a language for any

3. {e} is a language for any

4.
5.

. Note that,
. Because the language F does not
contain any string but {e} contains one string of length zero.
The set of all strings over { 0, 1 } containing equal number of 0's and 1's.
The set of all strings over {a, b, c} that starts with a.

Convention : Capital letters A, B, C, L, etc. with or without subscripts are normally used to
denote languages.

Set operations on languages : Since languages are set of strings we can apply set
operations to languages. Here are some simple examples (though there is nothing new in it).

Union : A string

iff

or

Reversal of a language :
The reversal of a language L, denoted as

, is defined as:

Example :

1.
2.

Let L = { 0, 11, 01, 011 }. Then

Let L = {

= { 0, 11, 10, 110 }.

| n is an integer }. Then

= {

| n is an integer }.

Language concatenation : The concatenation of languages

= { xy |

and

}.

and

is defined as

Example : { a, ab }{ b, ba } = { ab, aba, abb, abba }.

Note that ,

1.
2.

in general.

3.
Iterated concatenation of languages : Since we can concatenate two languages, we also
repeat this to concatenate any number of languages. Or we can concatenate a language with
itself any number of times. The operation
This is defined formally as follows:

denotes the concatenation of L with itself n times.

Example : Let L = { a, ab }. Then according to the definition, we have

Kleene's Star operation : The Kleene star operation on a language L, denoted as

is

defined as follows :
= ( Union n in N )
=
= { x | x is the concatenation of zero or more strings from L }
Thus
is the set of all strings derivable by any number of concatenations of strings in L. It is also
useful to define
=

, i.e., all strings derivable by one or more concatenations of strings in L. That is

=
=

Example : Let L = { a, ab }. Then we have,

=

(Union n in N and n >0)

= {e}

{a, ab}

{aa, aab, aba, abab}

=
= {a, ab}

Note : e is in

{aa, aab, aba, abab}

, for every language L, including .

The previously introduced definition of

is an instance of Kleene star.

Automata and Grammars

An automata is an abstract computing device (or machine). There are different varities of such
abstract machines (also called models of computation) which can be defined mathematically.
Some of them are as powerful in principle as today's real computers, while the simpler ones are
less powerful. ( Some models are considered even more powerful than any real computers as
they have infinite memory and are not subject to physical constraints on memory unlike in real
computers). Studying the simpler machines are still worth as it is easier to introduce some
formalisms used in theory.
Every automaton consists of some essential features as in real computers. It has
a mechanism for reading input. The input is assumed to be a sequence of
symbols over a given alphabet and is placed on an input tape(or written on an
input file). The simpler automata can only read the input one symbol at a time
from left to right but not change. Powerful versions can both read (from left to
right or right to left) and change the input.
The automaton can produce output of some form. If the output in response to an
input string is binary (say, accept or reject), then it is called an accepter. If it
produces an output sequence in response to an input sequence, then it is called
a transducer (or automaton with output).
The automaton may have a temporary storage, consisting of an unlimited number
of cells, each capable of holding a symbol from an alphabet ( whcih may be
different from the input alphabet). The automaton can both read and change the
contents of the storage cells in the temporary storage. The accusing capability of
this storage varies depending on the type of the storage.

The most important feature of the automaton is its control unit, which can be in any one of
a finite number of interval states at any point. It can change state in some defined manner
determined by a transition function.

Figure 1: The figure above shows a diagramatic representation of a generic automation

Operation of the automation is defined as follows.

At any point of time the automaton is in some integral state and is reading a particular
symbol from the input tape by using the mechanism for reading input. In the next time step
the automaton then moves to some other integral (or remain in the same state) as defined
by the transition function. The transition function is based on the current state, input symbol
read, and the content of the temporary storage. At the same time the content of the storage
may be changed and the input read may be modifed. The automation may also produce
some output during this transition. The internal state, input and the content of storage at
any point defines the configuration of the automaton at that point. The transition from one
configuration to the next ( as defined by the transition function) is called a move. Finite
state machine or Finite Automationis the simplest type of abstract machine we consider.
Any system that is at any point of time in one of a finite number of interval state and moves
among these states in a defined manner in response to some input, can be modeled by a
finite automaton. It doesnot have any temporary storage and hence a restricted model of
computation.

Grammar
A grammar is a mechanism used for describing languages. This is one of the most simple
but yet powerful mechanism. There are other notions to do the same, of course.
In everyday language, like English, we have a set of symbols (alphabet), a set of words
constructed from these symbols, and a set of rules using which we can group the words to
construct meaningful sentences. The grammar for English tells us what are the words in it
and the rules to construct sentences. It also tells us whether a particular sentence is wellformed (as per the grammar) or not. But even if one follows the rules of the english
grammar it may lead to some sentences which are not meaningful at all, because of
impreciseness and ambiguities involved in the language. In english grammar we use many
other higher level constructs like noun-phrase, verb-phrase, article, noun, predicate, verb
etc. A typical rule can be defined as
< sentence > < noun-phrase > < predicate >
meaning that "a sentence can be constructed using a 'noun-phrase' followed by a
predicate".

Some more rules are as follows:

< noun-phrase >

< article >< noun >

< predicate >

< verb >

with similar kind of interpretation given above.

If we take {a, an, the} to be <article>; cow, bird, boy, Ram, pen to be examples of <noun>; and
eats, runs, swims, walks, are associated with <verb>, then we can construct the sentence- a cow
runs, the boy eats, an pen walks- using the above rules. Even though all sentences are wellformed, the last one is not meaningful. We observe that we start with the higher level construct
<sentence> and then reduce it to <noun-phrase>, <article>, <noun>, <verb> successively,
eventually leading to a group of words associated with these constructs.
These concepts are generalized in formal language leading to formal grammars. The word
'formal' here refers to the fact that the specified rules for the language are explicitly stated in
terms of what strings or symbols can occur. There can be no ambiguity in it.

Formal definitions of a Grammar

A grammar G is defined as a quadruple.

N is a non-empty finite set of non-terminals or variables,

is a non-empty finite set of terminal symbols such that
,

is

special

and

(or

variable)

called

the

start

is

denoted

symbol,

is a finite set of production rules.

The

binary

i.e.
In

non-terminal

relation

defined

iff
other

where

words,

by

the

set

of

production

rules

by

.
P

is

finite

set

of

production

rules

of

the

form

and

Example : Consider the grammar

, where N = {S},

={a, b} and P is the set of

the following production rules

{S

ab, S

aSb}

Some terminal strings generated by this grammar together with their derivation is given below.

ab

aSb

aabb

aSb

aaSbb

aaabbb

It is easy to prove that the language generated by this grammar is

By using the first production, it generates the string ab ( for i =1 )

Finite Automata

Finite Automata
Automata (singular : automation) are a particularly simple, but useful, model of computation. They
were initially proposed as a simple model for the behavior of neurons. The concept of a finite
automaton appears to have arisen in the 1943 paper A logical calculus of the ideas immanent in
nervous activity", by Warren McCullock and Walter Pitts. In 1951 Kleene introduced regular
expressions to describe the behaviour of finite automata. He also proved the important theorem
saying that regular expressions exactly capture the behaviours of finite automata. In 1959, Dana
Scott and Michael Rabin introduced non-deterministic automata and showed the surprising
theorem that they are equivalent to deterministic automata. We will study these fundamental
results. Since those early years, the study of automata has continued to grow, showing that they
are indeed a fundamental idea in computing.

Finite Automata :
A finite automaton has: Finite set of states, with start/initial and accepting/final states; Transitions
from one state to another on reading a symbol from the input

States, Transitions and Finite-State Transition System :

Let us first give some intuitive idea about a state of a system and state transitions before
describing finite automata.
Informally, a state of a system is an instantaneous description of that system which gives all
relevant information necessary to determine how the system can evolve from that point on.
Transitions are changes of states that can occur spontaneously or in response to inputs to the
states. Though transitions usually take time, we assume that state transitions are instantaneous
(which is an abstraction).
Some examples of state transition systems are: digital systems, vending machines, etc.

A system containing only a finite number of states and transitions among them is called a finitestate transition system.
Finite-state transition systems can be modeled abstractly by a mathematical model called finite
automation.
We said that automata are a model of computation. That means that they are a simplified
abstraction of `the real thing'. So what gets abstracted away? One thing that disappears is any
notion of hardware or software. We merely deal with states and transitions between states. The
distinction between program and machine executing it disappears. One could say that an
automaton is the machine and the program. This makes automata relatively easy to implement in
either hardware or software. From the point of view of resource consumption, the essence of a
finite automaton is that it is a strictly finite model of computation. Everything in it is of a fixed, finite
size andcannot be modified in the course of the computation.

Deterministic Finite Automata (DFA)

Informally, a DFA (Deterministic Finite State Automaton) is a simple machine that reads an input
string -- one symbol at a time -- and then, after the input has been completely read, decides
whether to accept or reject the input. As the symbols are read from the tape, the automaton can
change its state, to reflect how it reacts to what it has seen so far.
Thus, a DFA conceptually consists of 3 parts:
1. A tape to hold the input string. The tape is divided into a finite number of cells. Each cell
holds a symbol from .
2. A tape head for reading symbols from the tape
3. A control , which itself consists of 3 things:
o finite number of states that the machine is allowed to be in (zero or more states are
designated asaccept or final states),
o a current state, initially set to a start state,
o a state transition function for changing the current state.

An automaton processes a string on the tape by repeating the following actions until the tape
head has traversed the entire string:
1. The tape head reads the current tape cell and sends the symbol s found there to the
control. Then the tape head moves to the next cell.
2. he control takes s and the current state and consults the state transition function to get the
next state, which becomes the new current state.
Once the entire string has been processed, the state in which the automation enters is examined.
If it is an accept state , the input string is accepted ; otherwise, the string is rejected . Summarizing
all the above we can formulate the following formal definition:

Deterministic Finite State Automaton : A Deterministic Finite State Automaton (DFA) is a 5-

tuple :

Q is a finite set of states.

is the next state transition function (which is total ). Intuitively,

function that tells which state to move to in response to an input, i.e., if M is in

is a finite set of input symbols or alphabet.

state q and sees input a, it moves to state

is the start state.

is the set of accept or final states.

is a

Acceptance of Strings :
A DFA accepts a string
that
1.
2.

if there is a sequence of states

in Q such

is the start state.

for all

3.

Language Accepted or Recognized by a DFA :

The language accepted or recognized by a DFA M is the set of all strings accepted by M ,
and is denoted by
i.e.
The notion of acceptance can also be made more precise by extending the transition
function .

Extended transition function :

Extend
.

(which is function on symbols) to a function on strings, i.e.

That is,
is the state the automation reaches when it starts from the state q and finish
processing the string w. Formally, we can give an inductive definition as follows:
The language of the DFA M is the set of strings that can take the start state to one of the accepting
states i.e.

L(M) = {
={

| M accepts w }
|

Example 1 :

is the start state

It is a formal description of a DFA. But it is hard to comprehend. For ex. The language of the DFA
is any string over { 0, 1} having at least one 1

We can describe the same DFA by transition table or state transition diagram as
following:

Transition Table :
0

It is easy to comprehend the transition diagram.

Explanation : We cannot reach find state

The self-loop at
string.

w/0 or in the i/p string. There can be any no. of 0's at the beginn

on label 0 indicates it ). Similarly there can be any no. of 0's & 1's in any order at the end o

State transition diagram :

Transition table :

It is basically a tabular representation of the transition function that takes two arguments (a state and a symbol
returns a value (the next state).

Rows correspond to states,

Columns correspond to input symbols,
Entries correspond to next states
The start state is marked with an arrow
The accept states are marked with a star (*).

(State) Transition diagram :

A state transition diagram or simply a transition diagram is a directed graph which can be constructed as follows:
1. For each state in Q there is a node.
2. There is a directed edge from node q to node p labeled a iff
that cause a transition, the edge is labeled by the list of these symbols.)
3. There is an arrow with no source into the start state.
4. Accepting states are indicated by double circle.

. (If there are several input sym

Here is an informal description how a DFA operates. An input to a DFA can be any string
. Put a pointer t
start state q. Read the input string w from left to right, one symbol at a time, moving the pointer according to the

transition function, . If the next symbol of w is a and the pointer is on state p, move the pointer to
. Wh
the end of the input string w is encountered, the pointer is on some state, r. The string is said to beaccepted by th
DFA if
A language

and rejected if

. Note that there is no formal mechanism for moving the pointer.

is said to be regular if L = L(M) for some DFA M.