TELUGU OCR FRAMEWORK USING DEEP LEARNING

By Rakesh Achanta* , and Trevor Hastie*

Stanford University*
Abstract: In this paper, we address the task of Optical Character
Recognition(OCR) for the Telugu script. We present an end-to-end
framework that segments the text image, classies the characters and
extracts lines using a language model. The segmentation is based on
mathematical morphology. The classication module, which is the
most challenging task of the three, is a deep convolutional neural
network. The language is modelled as a third degree markov chain at
the glyph level. Telugu script is a complex abugida and the language
is agglutinative, making the problem hard. In this paper we apply the
latest advances in neural networks to achieve acceptable error rates.

1. Introduction. There has been limited study in the development of

an end-to-end OCR system for the Telugu script. While the availability of a
huge online corpus of scanned documents warrants the necessity for an OCR
system, the complex script and agglutinative grammar make the problem
hard. Building a system that works well on real-world documents containing
noise and erasure is even more challenging.
The task of OCR is mainly split into segmentation and recognition. The
design of each is guided by that of the other. The more robust (to noise,
erasure, skew etc.) the segmentation is, the easier the task of the recognizer
becomes, and vice-versa. The techniques used in segmentation are somewhat
similar across scripts. This is because, usually, one connected component (a
contiguous region of ink) can be extracted to give one unit of written text.
While this principle applies to the Roman scripts with few exceptions; it
does not hold for complex scripts like Devanagari and Arabic, where words,
not letters, are written in one contiguous piece of ink. The Telugu script is
of intermediate complexity, where consonant-vowel pairs are written as one
unit.
The recognition task is traditionally split into feature extraction and clas-
sication. The former has been hand-engineered for a very long time. As
early as 1977, Telugu OCR systems used features that encode the curves that
trace a letter, and compare this encoding with a set of predened templates
Keywords and phrases: Machine Learning, Deep Learning, Convolutional Neural Net-
works, Optical Character Recognition, Gradient Based Learning, Document Recognition,
Telugu, Natural Language Processing

1
2 R. ACHANTA ET AL.

Rajasekaran and Deekshatulu (1977). This same approach was followed later
in Rao and Ajitha (1995). The rst attempt to use neural networks for Tel-
ugu OCR to our knowledge was in Sukhaswami, Seetharamulu and Pujari
(1995). They train multiple neural networks, and pre-classify an input image
based on its aspect ratio and feed it to the corresponding network. This re-
duces the number of classes that each sub-network needs to learn. But this is
likely to increase error rate, as failure in pre-classication is not recoverable.
The neural network employed is a Hopeld net on a down-sampled vector-
ized image. Later work on Telugu OCR primarily followed the featurization-
classication paradigm. Combinations like ink-based features with nearest
class centroid Negi, Bhagvati and Krishna (2001); ink-gradients with near-
est neighbours Lakshmi and Patvardhan (2002); principal components with
support vector machines(SVM) Jawahar, Kumar and Kiran (2003); wavelet
features with Hopeld nets Pujari et al. (2004) were used. More recent work
in this eld Kumar et al. (2011) is centred around improving the supporting
modules like segmentation, skew-correction and language modelling.
This paper improves on previous work in a few signicant ways. While
previous work was restricted to using only a handful of fonts, we develop
a robust font-independent OCR system by using training data from fty
fonts in four styles. This data (along with the rest of the OCR program)
is publicly released to act as a benchmark for future research. The training
and test data are big and diverse enough to let one get reliable estimates
of accuracy. Our classier achieves near human classication rate. We also
integrate a much more advanced language model, which also helps us recover
broken letters.
In our work, we break from the above mentioned featurize and classify
paradigm. We employ a convolutional neural network(CNN), which learns
the two tasks in tandem. In addition, a CNN also exploits the correlation
between adjacent pixels in the two dimensional space LeCun et al. (1998).
Originally introduced for digit classication (a sub-task of OCR), CNNs
have been adapted to classify arbitrary colour images from even a thousand
classes Krizhevsky, Sutskever and Hinton (2012). This was aided in part
by better regularization techniques like training data augmentation Simard,
Steinkraus and Platt (2003), dropout Hinton et al. (2012) and by increased
computing power. We take this opportunity to go on an excursion into deep
learning, and try to demystify the process of training big and deep neural
networks. We not only solve the image classication problem at hand but
also integrate it into the larger framework for text recognition.
The rest of the paper is organized as follows. We introduce the problem
in Section 2, describing the Telugu language and its calligraphy. Section
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 3

3 explains the segmentation process and Section 4 generation of training

data. Section 5 describes CNNs in complete detail, with a subsection (5.3)
on our architecture. Results are discussed in 5.4 and regularization in 5.6.
The language model, including the recovery of broken letters, is described
in Section 6. We conclude with an evaluation of the end-to-end system in
Section 7.

2. The Problem. Telugu is a Dravidian language with over 80 million

speakers, mainly in the southern India state of Andhra Pradesh. It has
a strong consonant-vowel structure, i.e. most consonants are immediately
followed by a vowel. A consonant and a vowel combine to give a syllable.
For example the syllable ka in the word sakarma in Figure 1 is one such
entity, and ra in the same gure is another. Each such syllable is written
as one contiguous ligature. This alphasyllabic form of writing is called an
abugida as opposed to an alphabet. There are 16 vowels and 37 consonants
which combine to give over 500 simple syllables. Of them about 400 are
commonly used. There are nearly sixty other symbols including vowel-less
consonants (m in Figure 1), punctuation and numbers. This brings the total
number of symbols used in common writing to approximately 460.
Our goal is to develop an end-to-end system that takes a page of Telugu
text and converts it to Unicode text. This would mean that we rst need
to detect lines in the image. Once the lines are detected, we need to further
segment each lines sub-image into individual letters. In this paper, we will
refer to the extracted connected components as glyphs. Our denition of a
glyph is one contiguous segment of ink, which typically maps to one syllable,
like ka and ra in Figure 1. While a syllable can be composed of more than
one connected component, like sa in the same gure, we can think of a
glyph as the visual for a syllable. After a glyph is obtained, it needs to be
recognized as representing one of the 460 classes. Once the candidates for
each glyph are obtained, we infer the most likely line of text from them using
a language model.

Calligraphy. The consonant forms the heart of the syllable and is written
at the center, between the top and bottom lines. The consonant is augmented
by a vowel which is written as the ascender. As an example let us consider
the moderately complex word sakarma rendered in Figure 1. While ka is
rendered as one glyph, rma is written in two. ra is written at the center
and a vowel-less m is written below the baseline disconnected from ra. A few
vowels like pa and sa have the vowels detached from the base.
4 R. ACHANTA ET AL.

Figure 1. The word (sakarma) rendered in Telugu and in English with Telugu styling.
The 3 in Telugu corresponds to the vowel a, usually attached to the consonant. The rst
syllable (sa) is written in two parts, (ka) in one, and (rma) is split into (ra) and
(m).

3. Segmentation. Given a binary image of text, where ink is unity

and background is zero, we rst nd its row-ink-marginal, i.e. the running
count of number of pixels that are ON. The image is skew corrected to
maximize the variance of the rst dierential of this marginal. That is, we
nd the rotation that results in the most sudden rises and falls in the row-
pixel-counts. Next we estimate the number of lines in the page. This is done
by taking the Discrete Fourier Transform of the mean-centred marginal. The
wavelength of the harmonic with the highest amplitude gives us an estimate
of the distance between two baselines. We use it as a heuristic to look for
sudden drops in the row-ink-marginal and tag them as baselines. We then
detect the zero between two baselines and identify them as line separations.
Additionally we also identify the toplines as the point of highest gain in
pixel count between the line-separation and the baseline. Thus the image is
separated into sub-images of lines. Figure 2 shows an example. Notice that
our algorithm is robust to descenders that are completely below the baseline.
Each line is processed to extract connected components using Leptonica
Bloomberg (2007), an image-processing library. We thus have, as input to
the classier, the arbitrary-sized glyph and its location in reference to the
base and top lines as in Figure 2. Figure 3 shows some glyphs ready to be
classied.

4. Generating the Training Data. To train our glyph classier, we

need training data. As there is no publicly available data for this task, we
set out to generate our own in a mechanized and inexpensive manner. We
assemble a moderate sized corpus (150 MB) of unicode Telugu text from
the internet. From this we extract the unigram counts of individual glyphs,
bigram counts of pairs and trigram counts of triples. For the task of generat-
ing sample training images, we only consider the most common glyphs. The
support is limited to glyphs that have a frequency of at least 104 . These
glyphs are rendered in fty dierent fonts in four dierent styles (normal,
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 5

Figure 2. A sample Telugu poem. The detected line-separations are in red, top-lines in
blue and bottom-lines in green. The marginal and the best tting harmonic are shown to
the right. Lines are detected by looking for sudden jumps in the marginal and using the
wavelength of the harmonic as a heuristic.

Figure 3. Glyphs extracted from a text image. Each connected component, shown in a
dierent colour, is scaled before being fed to the Neural Network.

bold, italic and bold-italic). This results in nearly 160 unique rendering per
glyph (after removing duplicates). Hence in all, we have 160 460 73, 000
labelled samples. Figure 5 shows all the renderings for a sample class. We
think these cover the range of modern type-settings in Telugu very well.
However, we still could not nd the computer equivalent for some of the
main fonts used extensively in the early days of printing. This leaves us
vulnerable to the problem of data drift, where the distribution of the train-
ing and test datasets does not completely capture that of the real world
problem. We need to consider this while regularizing our classier.
6 R. ACHANTA ET AL.

Figure 4. The word jarjara. The The letter (ja) when rendered above the base line has
its usual interpretation. But when rendered below, it results in the loss of a vowel here a.

Some glyphs can not be rendered stand-alone since they exist only as
parts of multi-glyph syllables. For e.g., the consonant combiner (+m) in
Figure 1 can only be a part of a syllable like (rma). So we render random
text on to a digital image and extract the glyphs back. Since we know the
rendered text, we know the labels of the glyph segments obtained. More
importantly, we also know the location information for each glyph, i.e. its
position relative the top line and baselines. We expect to use this informa-
tion for better classication. Since the same glyph, when placed at dierent
locations relative to the top and base lines, results in it being in dierent
classes (Figure 4 shows an example). Now we have the training data that
can be used to train any machine learning algorithm of interest.

5. Convolutional Neural Networks. Convolutional Neural Networks

(CNNs) have been shown to be very successful for image recognition tasks
over a broad spectrum of problems from digit recognition LeCun et al. (1998)
to the ImageNet classication Krizhevsky, Sutskever and Hinton (2012), and
to the more recent Chinese hand-written character recognition task Cirean
and Schmidhuber (2013). This motivates us to use CNNs for the Telugu
recognition task. But our problem diers from digit recognition in that it
has many more classes (460 as opposed to 10). It also diers from the Chi-
nese OCR task in that we need to incorporate location information into our
classier (Figure 4).

5.1. The Architecture. CNNs were introduced LeCun et al. (1998) to

exploit the two-dimensional correlation structure of image data. A vanilla
neural network which ignores this structure (by attening the input image)
is not well suited for such pattern recognition tasks. The convolutional and
pooling layers that operate on two or three dimensional image data are at
the heart of the CNN architecture.
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 7

Figure 5. The 167 unique renderings of the letter (go). This illustrates the expected
spread in the distribution of glyphs. Similarly we have nearly as many unique renderings
for each of the 460 classes.

5.1.1. Convolutional Layer. In full generality, input to a CNN is a 3D-

image with two spatial dimensions and one frequency dimension. For colour
images, each sample is a stack of three maps; one each for red, blue and
green. Hyper-spectral images could have many more, while gray-scale images
have only one map. In our case, the input is a 48 48 single-map binary
image. Given that the convolution operation employed by CNNs is very
complicated, we begin by explaining a simple convolution operation on a 2D
image.

2D-Convolution. A 2D-convolution operation is parametrised by a 2D-

kernel of size, say, k = 2l + 1. Typically k is set to three or ve (however,
for smaller input images, it could be set to even two). At each pixel in
the interior of the input image, this kernel is applied. That is, we take
the dot-product of the kernel with the l-neighbourhood of the target pixel.
Convolution with a random kernel thus performs a local weighted-averaging
over the input image. More useful kernels tend to act as various kinds of
edge/feature detectors. Convolution with a 2l + 1 kernel reduces the side of
the image by l, which might lead to loss of information at the borders. To
avoid this, the original image can be zero-padded by l pixels. Figure 6 shows
8 R. ACHANTA ET AL.

a few such kernels applied to a binary image.

Figure 6. The original binary image of the letter (ja) is shown at the top, the three
sample convolution kernels at the center, and the corresponding outputs at the bottom.

3D-Convolution. A 3D image is a stack of 2D images (which are referred

to as maps in this context). In CNNs, intermediate representations of the
input image are in fact 3D images with possibly thousands of maps. These
maps tend to be of a smaller size than that of the input.
As shown in Figure 7, each constituent map of the 3D input image is con-
volved with its corresponding 2D kernel. The dot-products are then summed
to generate a single output map. Hence an input with din maps gets con-
volved with a 3D kernel of size k k (and depth also din ) to give a single
output map. Where dout output maps are required, we employ as many
such kernels. Thus a 3D to 3D convolution operation has a kernel of size
dout din k k. In all, the 3D convolution kernel has k 2 din dout param-
eters. We can think of this operation as a series of dout feature extractors.
The convolution operation CW on a 3D input tensor A (of size d s s)
in
with a kernel W (of size k = 2l + 1) can be written as:

(5.1) CW : Rdin ss Rdout ss

din
l
l
z
(5.2) CW (A)z,x,y = Wm,i,j Am,x+i,y+j
m=1 i=l j=l

with the assumption that for any indices out of range, A is zero. A non-
to give the nal C
linearity R is applied after the convolution CW W operation
of the convolutional layer.

(5.3) CW (A) = R(CW (A))
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 9

+ +

... ... ...

Figure 7. An input color image, its RBG maps, the convolutional kernels and the output
maps. Many such output maps are generated to form the 3D output image. Note, however,
that the maps lose the RGB interpretation after the rst layer.

Here, W is the weight or parameter tensor that is to be learned by training

the network. We explain the role of non-linearities shortly; for now they can
be thought of as a thresholding to identify interesting features.

5.1.2. Pool Layer. A typical convolution layer outputs many more maps
than it takes in. There is also high correlation between the adjacent output
values in a map. Hence it makes sense to somehow scale the maps down,
generally by a factor of two in each co-ordinate. This operation is called
pooling. Typically pooling is done over a 2 2 grid. The maximum of the
four pixels in the grid is extracted as output. Taking maximum (as opposed
to the average) gives the neural network translational invariance, which is
key to good classication. We can increase the number of maps four-fold
at each convolutional layer and decrease the area of the image-maps by a
factor of four at the succeeding pool layer. This preserves the size of the
image while at the same time transforming it to a dierent, more useful,
10 R. ACHANTA ET AL.

space. The 2 2 max-pooling operation on a tensor A can be written as

(5.4) P2 (A)z,x,y = max Az,i,j

i=2x1,2x
j=2y1,2y

5.1.3. Fully-connected Layer. CNNs can have anywhere between two to

twenty convolutional and pooling layers. Their nal output, which is a 3D
image, is attened into a vector. One or more fully-connected layers follow.
A fully-connected layer is basically a simple matrix multiplication followed
by a non-linearity.

(5.5) F W : Rn1 Rn2

(5.6) FW (A) = R(W A)

Here again W is the weight matrix that is to be learned by training the

network, and R is a general non-linearity (usually applied element-wise).
Note that this is nothing but the typical hidden layer of a traditional neural
network.

5.1.4. Output Layer. The last of such fully-connected layers is the output
layer FWS . It has as many nodes as the number of classes, K, in the problem.

It employs a specic form of non-linearity, the softmax function S.

(5.7) S
FW : Rn2 RK
S
(5.8) FW (A) = S(W A)

where,

eAk
(5.9) Sk (A) = K
Aj
j=1 e

This is the same transformation used in the multilogit model, and produces
positive values that sum to one. The K-vector obtained from the matrix
multiplication is exponentiated to make it all positive and then normalized
to belong to the K-simplex. These K values are interpreted as the class
probabilities. In summary, one can think of the convolutional and pool layers
as the feature extraction phase of the CNN and the the fully connected layers
as a vanilla neural network classier on top.
Let X, Y denote a random image and its class label respectively. Then,
according to our model, the likelihood is given by

(5.10) P (Y = y | X = x; W) = py (x; W) = Sy (W1 A(x))

 TELUGU OCR FRAMEWORK USING DEEP LEARNING 11

where A(x) is the input to the softmax layer (for a given set of network
parameters W = {Wi }). Neural networks are trained to maximize the log of
the above likelihood over the training data D to nd the optimum network
parameters W .

(5.11) L= log (py (x; W))
(x,y)D

(5.12) W = argmax L

W

5.2. Non-linearities. Non-linearities are applied to the output of each

layer of the neural network. A neural network with one non-linear hidden
layer can act as a universal function approximator Cybenko (1989), i.e. it
can approximate any continuous function on a compact space (given enough
nodes). The Sigmoid function has had been the most commonly used non-
linearity. It and the closely related hyperbolic tangent function resemble step
functions.
1
(5.13) sigmoid(x) =
1 + ex
ex ex
(5.14) tanh(x) = x = 2 sigmoid(2x) 1
e + ex
Each sigmoid node in the hidden layer simulates a step function of a dif-
ferent size at a dierent location. The output node combines these steps
to approximate the desired mapping. Without the non-linear activations, a
neural net, regardless of depth, can only act as a linear transformer.
In recent years rectier linear units(ReLU) have gained in popularity Nair
and Hinton (2010). A rectier is just the x+ function, in a sense it is the
most basic non-linearity.
(5.15) ReLU(x) = x+ = max(0, x)
Despite their simplicity and zero gradient in half of the domain, they im-
prove performance of our network in a signicant way. However, a rectier
activated network needs multiple starts (around ten in our case) to get a
successful training instance. This is because of a propensity for the gradients
to either blow up in the linear region or die down in the at region. It has
been observed that performance can be further improved by allowing the
rectier to leak a bit. That is, instead of being zero for negative values, it
has a small slope.
(5.16) x = max(0, x)
(5.17) LeakyReLU(x) = x+ x
12 R. ACHANTA ET AL.

leaky rectifier

1
rectifier
sigmoid
tanh

0
1

1 0 1

Figure 8. The four most common neural network activations.

The choice of is arbitrary. Experiments to learn the amount of leakage

He et al. (2015) demonstrate that it is desirable to decrease as we move
from input to output layers. For our network, instead of trying to learn
the leakages, we x them as follows. We start with = .5 for the rst
convolutional layer and decrease it at each layer to nally end with an =
.05 for the last hidden layer. Figure 8 shows the various non-linearities and
Table 2 summarizes the eect of these choices on classication error for our
application.

12 6
48
24

2
50
48

45

0
7

3
3
2
connect
convolve pool convolve pool convolve pool fully

Figure 9. A traditional convolutional neural network where a single channel 4848 input
is subjected to three convolution-pool operations (resulting in 6 24 24, 16 12 12,
30 6 6 maps respectively). The nal 30 6 6 tensor is attened to 1080 nodes and
is fully connected to a hidden layer with 500 units, followed by a multi-class logistic layer
over the 457 classes.
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 13

5.3. Our Neural Network. The glyphs obtained from segmentation are
scaled up or down to a 48 48 square and fed to the CNN. We preserve
the aspect ratio while scaling. In addition to these 2304 binary pixel values,
we have two numbers representing the locations of the top and baselines,
which we will later incorporate into the network. We use a traditional ar-
chitecture, based on LeCun et al. (1998), as our reference point to compare
various design choices and regularizations. It has three pairs of convolutional-
pool(conv-pool) layers, all of which employ a 3 3 convolution kernel and a
2 2 max-pooling window. These are followed by two fully connected layers.
We use leaky-ReLUs in place of the tanh activations of LeCun et al. (1998).
The last layers soft-max activation yields the nal class probabilities. This
model can can be written as.

(5.18) p(x; W) = FW
S
5
FW4 P2 CW3 P2 CW2 P2 CW1 (x)

W {Wi } are the network parameters to be learned. Figure 9 shows the

traditional model and Figure 10 shows the output at each layer for a sample
input. Table 1 summarizes error rates, sizes and speeds of dierent architec-
tures.
Table 1
Performance of various architectures. All networks employ input distortion as
regularization. 48 48 represents the binary input image, mC3 a 33 convolutional layer
with m output maps, MP2 a 22 max-pooling layer, mN a fully connected layer with m
units and 457SM a softmax output layer on 457 classes. Speed is specied in terms of
time taken to classify one image on a GPU. Network size is specied in millions of
parameters. The standard error of the error estimates is .1%

Architecture %Training %Test Speed Size

Error Error (ms) (M)
1 Ten nearest neighbour classier 26.40 35.55 10
2 Linear support vector machine 25.16 28.65 .023 1.05
3 Multi-class logistic (48x48-457SM) 29.08 32.06 .039 1.05
4 One Hidden Layer (48x48-1000N-457SM) 24.63 28.38 .057 2.76
5 48x48-2000N-2000N-457SM 09.20 13.00 .130 9.52
6 48x48-8C3-MP2-24C3-MP2-72C3-MP2- 00.58 01.48 .259 1.14
-500N-457SM (traditional)
7 48x48-6C3-6C3-MP2-18C3-18C3-MP2-54C3- 00.09 00.74 .646 0.75
-54C3-MP2-162C3-162C3-MP2-457SM
(deepest)
8 48x48-4C3-MP2-6C3-8C3-MP2-32C3-50C3- 00.41 01.36 .168 0.25
-50C3-MP2-457SM (slim)
14 R. ACHANTA ET AL.

5.4. Discussion of Results. The k-Nearest Neighbour classier does very

poorly on this data with a 35% test error. This justies the need to use
more sophisticated models. With multi-class logistic regression, the test er-
ror comes down to 32%. It further reduces to 27% with a simple linear
SVM, a more robust classier. Both of these have around a million pa-
rameters, which can be considered as a point of reference for model size. A
conventional neural network with two hidden layers, each with two thousand
nodes, takes the error down to 13% while using nearly 10 million parame-
ters. All these architectures ignore the 2D structure of the input data and are
not specialized for the problem at hand. A structure-exploiting CNN with
three conv-pool operations followed by two fully connected layers gives an
error rate of 1.48% with nearly as many parameters as a multi-class logistic
model. Deeper networks with more convolutional layers and only one fully
connected layer give better and near human accuracies for this problem,
while keeping the network size small.

Figure 10. The letter (go) goes through the slim network from Table 1. The last two
images (at the bottom right) are the input and output to the softmax classication layer.
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 15

Figure 11. Intermediate features and nal transformations for various input images (ex-
tracted from the fourth and the last hidden layers respectively of the slim network). Notice
how similar looking images have similar representations in the transformed space.

One can make the network size as small as 100K parameters while still
achieving test error rates below 1.5%. For these deeper networks (with only
the one fully connected layer) half of the parameters are in this last clas-
sication layer. The rest of the network with its six to eight convolutional
layers uses the other half of the parameters. Thus the network parame-
ters are equally split between the classication and feature-extraction tasks.
However, the convolution operation is computationally expensive and can
signicantly increase the time required to classify a glyph. But as we are in
the sub milli-second range, this is acceptable. Even the deepest neural net-
works, once trained, classify much faster than a kNN classier. Also, in the
context of the larger Telugu OCR problem, the Viterbi algorithm employed
by the language model is the main speed bottle neck.
While it might seem natural to use the Fast Fourier Transform(FFT) to
perform the convolution operation, it does not give any signicant improve-
ment in performance as the kernel size is small and xed at three. While
direct convolution needs 9n operations, FFT would require n log(n)+n oper-
ations, which is not a signicant reduction in complexity. FFT has additional
memory requirements to store the transformation; and back-propagating
through it is not as straightforward.

5.5. Design Choices. The fourteen layered neural network from Table 1
(line 7) requires the specication of nearly a hundred hyper-parameters.
It does take quite a bit of experimentation to obtain the set of hyper-
parameters that lead to good prediction. The slim architecture from Ta-
16 R. ACHANTA ET AL.

ble 1 has been carefully hand-crafted to improve both on size and speed
over the traditional architecture. One could spend countless hours tuning
these hyper-parameters. This diculty in picking the right architecture and
hyper-parameters gives neural networks an air of mystery. Here we will
briey point out some of the important design choices in an attempt to
demystify the process.
In our dataset of binary images, nearly one-fth of the pixels are ink and
the rest are background. If we consider ink to be one and background zero,
the mean of the image is at 20%. One can instead consider the reverse and
increase the mean to 80%. A higher mean will help keep the ReLU units
activated (i.e. in their linear, as opposed to at, region). This is a non-issue
for the leaky ReLUs, which are never really at. In addition to more success-
ful training instances, the leaky ReLUs also improve performance by about
25%. Using traditional tanh activation not only increases the computational
complexity but also reduces performance by 25%. Table 2 summarizes these
eects.
Table 2
The eect of various design choices on test and training errors. We use the traditional
architecture from line 6 of Table 1. We report median rates over eleven training attempts.
The standard error of the error estimates is .1%

Design %Training %Test

Choices Error Error
Traditional architecture 0.58 1.48
No input inversion 0.56 1.40
No leaky ReLUs (needs multiple starts) 0.97 2.00
tanh activations 1.10 2.04

The traditional CNN architecture, based on LeCun et al. (1998), has a

pool layer after each convolutional layer. The kernel size decreases as the size
of the image decreases with pooling. Modern architectures tend to use 3 3
kernels all through. Where bigger kernels are needed, multiple convolutional
layers are used instead. For e.g., a pair of back to back convolutional layers,
with kernel sizes of three each, together result in a receptive eld of size ve.
They need 2 3 3 = 18 weights in all, where as a convolutional layer with
a kernel size of ve needs 5 5 = 25 weights, and has one less non-linearity.
However, we pay the price with increased computational complexity.

5.6. Regularization. Given that we are tting a {0, 1}4848 (0, 1)457
model using over a million parameters, it is very easy to overt, making
regularization a key component of the training process. Two forms of regu-
larization seem to work well for this problem dropout Hinton et al. (2012)
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 17

and input distortion. Both of them distort the input to a classier, albeit in
very dierent ways.

Input distortion. Corrupting input data to stabilize predictions has been

employed by various machine learning techniques Abu-Mostafa (1990); Simard
et al. (1998). In addition, it can be shown that that adding gaussian noise
to the data results in an L2 -type regularization Bishop (1995). We go much
farther than gaussian noise and apply the following distortions to the input
image (all of them by a random amount): translation, elastic deformation
Simard, Steinkraus and Platt (2003), zoom, rotation and salt & pepper noise.
Although the original training data better approximates both the test and
real-world data, we notice that distortion proves to be an eective form of
regularization. The network is less prone to over-tting given that it does
not see the same exact sample twice. It is forced to learn invariant features
instead of memorizing image patches. Figure 12 shows some sample distor-
tions.

Figure 12. Three distortions applied to the seven original images in the top row. Each
distortion is a combination of a random amount of translation, rotation, zoom, elastic
deformation and noise.

Dropout. Dropout employs multiplicative noise, where each input is ei-

ther doubled or set to zero with equal probability. (More generally, an input
1
is set to zero with a probability of else it is multiplied by a factor of 1 .
Unless explicitly specied, is 1/2.) This multiplicative noise results in a
penalty that is data dependent Wager, Wang and Liang (2013), thus incor-
porating the distribution of input data into the regularization process. An
alternate view of dropout the one that inspired its invention Hinton et al.
18 R. ACHANTA ET AL.

(2012) is that we encourage the network to learn features that are useful
by themselves rather than features that are useful in the presence of one
another. By dropping out a random half of the features at each gradient
descent step, we are preventing co-adaptation of features. This helps gener-
alization error. In other words, we are very weakly training a huge number
of models and averaging them at test time.
While dropout can be applied at any layer of the CNN, it does not make
much sense to use dropout at the convolution layers. If dropout is applied
at the input layer, it is the same as salt & pepper noise. Dropout works
best when applied to the nally extracted features just before feeding them
to the softmax layer. We hypothesize that dropping out a feature (that has
been extracted by eight convolutional layers) is equivalent to distorting the
input image signicantly. Thus dropout and input distortions have similar
eects. This is also apparent in Table 3, which summarizes performance
using various forms of regularization.
Table 3
The eect of various forms of regularization on test and training errors. We take the
traditional architecture from Table 1 and remove one form of regularization from it to see
how the absence aects performance. Median rates over 11 trails are reported. The
standard error of the error estimates is .1%

Regularization %Training %Test

Variations Error Error
Traditional 0.58 1.48
Without input distortion 0.01 1.77
Without dropout 0.23 1.13
Without input distortion & dropout 0.01 2.78
Without depth (third conv-pool layer) 1.19 2.69
Without input distortion, dropout & depth 0.00 4.10
Without salt & pepper noise 0.29 1.02
Without just the elastic aspect of distortion 0.48 1.16

The traditional architecture from line 6 of Table 1 gives a test error of

1.48%. This employs both dropout and distortion. Without both of them,
the test error shoots up to 2.78%. Using just dropout it is 1.77%, using
just the distortions it is 1.13%. While this might seem strange that dropout
aects performance adversely in the presence of distortions, it is only because
together the two forms of regularization are too much. This motivates us to
examine the eect of using a lower dropout rate. Figure 13 summarizes these
results. In the absence of distortions, a dropout rate of around 50% works
best; in its presence little to no dropout is needed. Without salt & pepper
noise we obtain better error rate of 1%, but this leaves the model susceptible
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 19

to data-drift, as we expect to see some stray pixels in real-world data.

3.0
2.5
%Test Error Rate
2.0
1.5

Input Distortion
Absent
1.0

Present

0 20 40 60 80
%Dropout

Figure 13. The eect of dropout on test error. Each dot represents one training attempt.
Solid lines show median performance for a given dropout rate over eleven such attempts.
In the presence of input distortion, no to little dropout is required. In its absence, however,
40-80% helps.

5.7. Location Information. Recall that when we extract a glyphs im-

age, we also know its position relative to the top and baselines (Figure 4).
Feeding location information into the neural network improves the accuracy
of our traditional model from 1.41% to 0.81%. This is a signicant gain
of 42%. This gain is mostly from glyphs like (ja) in Figure 4 where the
two (ja)s are indistinguishable by the original network. However, there
is no straight-forward way to augment the input image with the two real
numbers representing the relative position of the top and baselines. In fact,
these two numbers are features that do not need further processing. There-
fore, we treat them as such and append them to the input of the output
layer (the output layer accepts a attened version of whatever the layer be-
fore it gives out). We should ensure that these values are at the same scale
as the rest of the features that are being fed to the softmax layer. Else, they
might fail to assert themselves since we apply some sort of L2 regularization
on the weights (via, dropout, distortions or early-stopping). Table 4 shows
performance gained by incorporating location information into the various
architectures.
20 R. ACHANTA ET AL.

Table 4
Improved test error with location information for networks from Table 1

Architecture %Without %With

Multi-class logistic 28.43 22.39
tanh activated traditional 02.04 01.64
traditional 01.42 00.82
deepest 00.74 00.56
slim 01.36 00.93

5.8. Implementation and Training. Given an image, the full model, as

specied by (5.18), gives the output probabilities. We can then calculate the
log-likelihood over the training data according to (5.11). The negative of the
log-likelihood(NLL) is the cost we wish to minimize along with a regular-
ization penalty, , on the weights. is further augmented by sophisticated
L2-equivalent penalties like dropout, input distortion and early stopping
(that are incorporated via the training process).

(5.19) cost = log (py (x; W)) + (W)
(x,y)D

While one could use the usual back-propagation algorithm to nd gradi-

ents of the cost with respect to the network parameters, we use the more
modern approach of symbolic dierentiation. The network parameters W
are initialized as symbols in the Python based software package Theano
Bastien et al. (2012); Bergstra et al. (2010). The nal cost is specied in
terms of these network parameters as given by the above equation. Theano
gives us the gradient of the cost with respect to the network parameters.
Traditionally back-propagation was used to eciently apply the chain-rule,
one layer at a time, to get the desired gradients. Symbolic dierentiation
packages apply the chain-rule internally to give us a gradient function ob-
ject that can be evaluated at a point of interest in the W space. In addition
to liberating us from having to write complicated back-propagating code,
Theano also leverages processing power of Graphics Processor Units (GPU)
available on most modern computers.
Once we have the gradient of our nal cost with respect to the network
parameters, we can perform stochastic gradient descent in the parameter
space. We use mini-batch stochastic gradient descent where gradients are
averaged over a mini-batch of 20 training samples. We also use momentum
or Nesterov averaging, where we step along a direction that is the exponen-
tially weighted running average of recent gradients (rather than along the
current gradient direction alone). The randomly initialized neural network
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 21

50.0
Test Error
Training Error
10.0
% Error

5.0
2.0
1.0
0.5

0 20 40 60 80 100

Pass

Figure 14. Training and Test error as a function of passes through the data.

trains for an hour, going over the entire dataset fty times, on a system with
Nvidia GTX Titan Black GPU, to give a test error rate of 2%. Training
is ve times slower without the GPU. We usually train the network with
100 passes over the training data. Convergence is faster with leaky ReLUs
than with ReLUs, as the former have gradients of larger magnitudes. For the
same reason, convergence is faster with ReLUs than with tanh activations.

6. Language Model. Once we have the list of probable candidates for

each extracted glyph, it remains for us to nd the most likely line of text. One
could just consider the most likely candidate for each glyph and output the
sequence of such best-matches as the nal extracted text. This might not be
the best thing to do, as there is a very strong dependency structure in human
speech and text. For this reason, while the CNN has a very good accuracy
on the training and test data, we might be able to further improve on it by
using a language model to aid classication. Using a language model helps
resolve any ambiguities that the CNN has between similar looking glyphs.
Spurious ink can attach two glyphs together while erasure can cut a glyph
into pieces. A language model can help us address both these problems.
22 R. ACHANTA ET AL.

Further, we can incorporate biases into the OCR system according to the
frequency of occurrence of various glyphs, letters, words, etc.
Given a sequence of input images x = (x1 , x2 , . . . , xt ) where xi {0, 1}4848 ,
we need to nd y = (y1 , y2 , . . . , yt ), the sequence of output labels that max-
imizes P (y|x).

(6.1) P (y|x) P (x|y)P (y)

First consider the term P (x|y). In reality there is a font based dependence
across glyph renderings {xi } given corresponding labels {yi }. But we make
the independence assumption that

(6.2) P (x|y) = P (xi |yi ).
i

While simplifying the math, it only makes the problem harder. Additionally,
it is applicable to our case as we do not incorporate font based dependencies
across glyphs in our neural network (say by maintaining a state as we move
from glyph to glyph). Now,

P (yi |xi )P (xi )

(6.3) P (xi |yi ) =
P (yi )

Now, P (xi ) drops out as it is does not depend on y. By substituting (6.2),

(6.3) in (6.1), we have,
P (yi |xi )
(6.4) P (y|x) P (y)
P (yi )
i

Here P (y) is the likelihood of a sequence of glyphs as a sentence in the

language. P (yi ) is the unconditional probability of a glyph in the language.
P (yi |xi ) is the probability of a class given an image. We do not know P (yi |xi )
but can approximate it by P (yi |xi ), which is the probability given out by
our classier when shown the image xi . These class probability estimates
P (yi |xi ), however, are learned by showing the network a dierent distri-
bution of P (yi ), denoted by P (yi ). The networks estimate of probabili-
ties P (yi |xi ) for a given image are biased by P (yi ). Hence we need to per-
form case-control correction to get the estimate P (yi |xi ). Assuming P (yi ) is
known, we have

P (yi )
(6.5) P (yi |xi ) P (yi |xi )
P (yi )
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 23

In our case, the training data contains equal number of instances per class
(although some classes are more than a thousand times as likely as some
others). Therefore, P (yi ) = 1/457 is the same for all i, hence it drops o
from (6.5). Now, plugging in P (yi |xi ) as an estimate for P (yi |xi ) in (6.4),
we obtain this simple formula as an estimate for P (y|x):

(6.6) P (y|x) P (y) P (yi |xi )
i

Here, P (y) is an estimate for P (y). Once we nd P (y) using the n-gram
model detailed below we nd the most probable sequence of glyphs as
y = argmaxy P (y|x).

6.1. n-gram Model. Although humans employ much more complicated

language models to infer from text, for the sake of character recognition
we will use a simple n-gram model. An n-gram model is an (n 1)th order
Markov model. That is, the current unit of language is independent of all but
the previous n 1 such units. A unigram model calculates the probability of
a sequence of units as the product of their individual probabilities. But this
seldom applies since we know, for example, a q is most likely followed by a u.
In English, lack of agglutination makes it possible to look up words, making
words the natural choice for language units. Telugu however, is agglutinative.
The sample expression from-in-between-those-two-persons should grammat-
ically be written as one word. This lack of a solid sense of word identity
is exacerbated by a further lack of standard convention for writing such
long words. The expressions from-in-between those-two-persons, from in-
between those-two-persons are also accepted in modern verse. In addition, as
the language is strongly phonetic, dierent accents of the same word should
be rendered dierently. Hence word level dictionaries are of limited use. We
use a glyph as a basic language unit.
The probability of a sentence is given thus by

t
(6.7) P (y) = P (yi |yi1 , yi2 , . . . , yi(n1) )
i=1

Where t is the number of glyphs in the sequence y. Here, by notation, y0

represents the beginning of a sentence and we ignore all yi with a negative
index. In our problem, we employ a trigram (second order markov) model
at the glyph level. Since a trigram is supported by 4573 108 points, it is
rich enough to stand in for a dictionary. Given the syllabic nature of writing
the language, it is impossible for some glyphs to occur after some other
24 R. ACHANTA ET AL.

glyphs. This renders the trigram incidence matrix sparse. We take the top
ve candidates for each glyph with corresponding scores P (yi |xi ) and run
the Viterbi algorithm to get the most likely sequence of glyphs according to
the formula

t
(6.8) P (y|x) P (yi |yi1 , yi2 )P (yi |xi )
i=1

which is obtained by substituting estimates for (6.7) in (6.6) with n = 3.

The estimates P (yi |yi1 , yi2 ) are learned from a digital corpus of Telugu
text.

6.2. Heuristic Over-segmentation. A major problem in scanned text is

erasure of ink. This results in glyphs being cut into two, like the m in Figure
15 appearing to be rn. Our neural network is trained only to recognize
whole glyphs. Even if a glyph is cut into parts, the neural network still
tries to classify the parts as whole glyphs. In this section we explain how to
broaden the scope of our problem to include text where some of the glyphs
are broken into two or more parts. The techniques presented in this section
can be easily applied to extending the scope of the problem in the other
direction, i.e., to recover glyphs that are joined together.
Our original segmentation algorithm works by looking for connected com-
ponents. Two pixels are said to be connected if one belongs to the four-
neighbourhood of the other. One could also consider eight neighbours in-
stead of four. Four-connection results in more broken glyphs than eight-
connection. But the latter has a higher propensity to mistakenly join two
glyphs. We employ four connection and try to rejoin broken glyphs. This
approach is called heuristic over-segmentation LeCun et al. (1998). The seg-
mentation module outputs a series of glyphs ordered from left to right (and
from top to bottom when the glyphs are approximately at the same ver-
tical location). These glyphs can be represented as a linear graph, where
each edge is a glyph. A set of n glyphs is a linear graph with n + 1 nodes.
Figure 15 shows an example.

Combining broken glyphs. The initial graph is linear and hence each
node has at most one child. The graph is parsed bottom up, starting from
the last node. At a given node, we consider the two consecutive edges from
self to child and from child to grandchild. We check to see if the two edges
need to be combined. This is repeated over all grandchildren of all children.
Although each node has only one child to begin with, each time we decide
to add a new edge, a grandchild of the current node becomes a child (like
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 25

r and n combining to give m in Figure 15). This way more than two pieces
can be combined to give a candidate glyph (like o, r, n combining to give
om in the same gure).

com
om
m
c o r n
Figure 15. Segmentation graph for a word that could be corn or com. Starting with the
black edges, given by the segmentation module, the green edges are to be added by our
recovery mechanism.

We currently use a set of ten heuristics to see if we should combine two

glyph-pieces. These fall under the following broad categories:
the top-match (of one or both the pieces) has low probability
various dimensions of the two glyphs are small
top-matches together have low bigram probability (or are forbidden)
there is considerable overlap between the boxes bounding the glyphs
We assign a weight to each rule, and if the total score exceeds a threshold, we
combine. These heuristics have been developed by looking at various failure
cases in sample texts. Alternately, one could use a sparse logistic regression
model and a large set of heuristically developed rules to get a subset of
important rules (and their corresponding weights). This needs additional
training data. It is better to over-combine than under, since both the original
pieces and their combination are considered for further processing. In our
example, both rn and m are considered by the nal Viterbi decoder.

6.3. Viterbi Decoder. Once we have our nal segmentation graph, which
is a Directed Acyclic Graph(DAG), we generate the recognition graph Le-
Cun et al. (1998). For each arc in the segmentation graph, we take the
corresponding image and pass it through our classier, and get the top M
candidates (and their probabilities). We build a new graph by replacing each
arc in the segmentation graph by M weighted arcs each carrying a dierent
candidate-character. It seems that M = 5 is sucient in practice. Figure 16
shows an example with M = 3 matches per image.
We now need to nd the path in the recognition graph that corresponds
to the highest probability as dened by (6.8). Note that (6.8) has two terms
26 R. ACHANTA ET AL.

m 25

# 15

% 15

c 75 o 85 r 85 n 85
e5 d5 t5 h5

u3 g4 l1 b2

m 95
w1
f1

Figure 16. Recognition graph for the image in Figure 15. Each arc is replaced by three
arcs corresponding to the top three matches for the image. Corresponding percentage prob-
abilities are also shown as edge weights.

per glyph: one is the probability of a candidate label given the image the
other is the n-gram probability of this candidate label given the previous
n 1. The former is incorporated in the recognition graph, however the
latter is not. Finding the strongest path in the recognition graph would
correspond to picking the top-match for each glyph. Doing so would not
incorporate any linguistic information. Hence we need to pick among all
the paths in the DAG, the one that has the highest path probability as
dened by (6.8). This could be computationally expensive. Suppose we have
a string of thirty glyphs with ve matches per glyph, we will have 530 (over
1020 ) paths to consider. To solve this seemingly intractable problem, we
use the famous Viterbi algorithm Viterbi (1967) an ecient dynamic
programming algorithm that nds the shortest path in a graph.
Before we can use the Viterbi algorithm, we need to augment the recogni-
tion graph with n-gram probabilities. The length of an edge is not just the
likelihood of a glyph on it, instead, we need to multiply it by the n-gram
probability of the current character-candidate given the candidates for the
previous n1. Each of the M edges (between a node and one of its children)
in the recognition graph is replaced by M n1 edges, where n is the order of
the n-gram. Thus, now, between a node and one of its children, we have M n
edges. We call this the n-gram graph. Figure 17 shows one such graph with
M = 2, n = 2. We run the Viterbi algorithm on this graph to get the most
likely path, which in our example from Figure 15 is corn.

6.4. Recalibrating the probabilities. Recall that to get an estimate of

P (yi |xi ), we correct the bias of P (yi |xi ) the probability given by our
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 27

cm 0,25

c# 0,15

em 5,15
e# 0,15

co 9,85 or 8,85 rn 3,85

_c 5,75 eo 1,85 dr 7,85 tn 0,85

_e 7,5 cd 0,5 ot 4,5 rh 0,5

ed 6,5 dt 0,5 th 6,5

om 2,95
dm 0,1

ow 1,1
dw 0,1

Figure 17. n-gram graph, shown here with two matches per image (M = 2) and bigram
probabilities (n = 2). For our application, we use M = 5, n = 3, leading to a much more
complicated graph. Viterbi algorithm nds the strongest path in this graph. The c,o in
the edge-label co 9,85 denote c as a candidate from the previous image and o from
the current. 85 is the %probability of o given the image and 9 is the %probability of the
bigram co. The edge strength is .09 .85 = .0765. _ represents beginning of a sentence.

classier. While under our model, P (yi |xi ) is asymptotically consistent for
P (yi |xi ), there are a few considerations. Firstly, a huge model like ours is
prone to over-t the values of P (yi |xi ) to the training data. While we do
regularize the network to prevent over-tting, we might have to further tune
the network parameters. This is because we are using the learned network as
a component in a bigger model that includes a language prior. To this end,
we recalibrate the learned probabilities by further penalizing the weights of
the softmax layer.
The class probability for the k th class is given by
T
ek A
(6.9) P (k|x) = ,
K jT A
j=1 e

where A is the input to the softmax layer (for a given image x) and {k }
dene the weight matrix W of (5.8). One could scale all the coecients j by
a xed quantity and the predictions of the neural network do not change,
28 R. ACHANTA ET AL.

only the condences in them do. Dene the biased probabilities P (k|x) as
T
ek A
(6.10) P (k|x) = .
K jT A
j=1 e

Here, multiplication by scales the norm of the vectors, thus acting as a

quadratic penalty. As 0, all classes will have almost-equal probabilities.
And as , almost all the probability mass will be concentrated on a
single class. For the CNN to work well with the n-gram model, it is important
to multiply the log-probabilities it gives by the correct amount of . Thus
becomes an additional tuning parameter for the combined model. In our
case, (.5, .75) seems to work well. If such a correction is not applied, the
n-gram model fails to assert itself in the face of overly-condent predictions
by the neural network. On the other hand, if is too small, we would be
biased towards outputting sentences that are highly likely according to our
language model (regardless of how the glyphs actually look like). As we use
the OCR system more, we can develop a better sense of what should be
for a given set of learned network parameters.

7. Comprehensive Evaluation. To test the integrity of the three

modules working together, we run an evaluation suite on six sample texts.
These texts, with varying levels of degradation, conform to the scope of our
problem to various levels. The results are summarized in Table 5 and the
sample texts are shown in Figure 18.
Table 5
Error-rates of the end-to-end system. The third column indicates if we have trained the
neural network on the exact or a similar font. The next column is the total number of
glyphs in the sample text; followed by the number of broken glyphs + the number of
instances where a glyph is attached to a neighbour. The last two columns are error rates
without and with the n-gram correction from the Language Model (LM).
Name of Language Known Glyph Broken + Errors Errors
the text Font Count Attached w/o LM w/ LM
a. Nudi Telugu Yes 119 01 (.8%) 3 (2.5%) 0 (0.0%)
b. Ksa Telugu No 155 29 (19%) 12 (7.7%) 4 (2.6%)
c. Annamaya Telugu No 132 11 (08%)+1 12 (9.1%) 5 (3.8%)
d. Nannayya Telugu Similar 129 33 (26%) 13 (9.3%) 7 (5.4%)
e. Bhrata Both Similar 125 25 (20%)+3 9 (7.2%) 7 (5.6%)
f. Rmayaa Sanskrit No 129 26 (20%) 12 (9.3%) 15 (11.6%)

For our evaluation we use the traditional architecture augmented with

location information (line 3 in Table 4). It uses 1.15M parameters to give a
test error of .81%. For the language model, we learn the trigram probabilities
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 29

(a)

(b)

(c)

(d)

(e)

(f)

Figure 18. The six sample texts used in the end-to-end evaluation.
30 R. ACHANTA ET AL.

from a modern corpus of 43M unicode Telugu characters obtained from the
internet. For this test, we do not re-calibrate the probabilities as described
in the previous section.
We get perfect recovery for the very well printed and scanned Nudi text
that uses a modern font. However, it is not a realistic specimen. To better
evaluate performance on our target documents, we test on fonts that have
not been seen by the neural network. It can be seen from Table 5 that the
network is not susceptible to data-drift, as the error-rates are not higher
for unseen fonts (of Ksa and Annamaya). Our implementation of the
recognition-graph also gives us signicant improvement in performance, as
it extends the scope of the problem to text with erasure of ink. We recover
well from a scenario where as high as one in four glyphs is broken (as in
Nannayya). We also see that implementation of the n-gram model further
reduces the error-rates for Telugu texts. However, it introduces more errors
for the Sanskrit text Rmayaa that is written in Telugu script. This is
understandable given that we use a corpus of modern Telugu to learn the
trigram probabilities. One could manipulate the recalibration parameter
(of Section 6.3) according to how similar the scanned text is going to be to
modern internet Telugu. All in all, we think our system gives satisfactory
performance over diverse looking documents. While this performance might
not be enough to replace a human transcriber, it is good enough to enable
search facility for a digital corpus of scanned texts.

8. Conclusion. We took up the challenging task of Telugu OCR and

have designed and implemented an end-to-end framework to solve it. This
system is very generalizable to other Brahmic alphasyllabic scripts like Kan-
nada, Malayalam, Gujarati, Punjabi, etc. the ones that do not use a
headline. Heuristics specic to the Telugu script played an important role in
solving the problem. This questions our frameworks generalizability. While
the heuristics used to detect lines are easily generalizable, the ones used to
combine glyphs need to be redesigned. We use the latest advances in deep
learning to design a very good classier, that is at the heart of the system.
We then use classical techniques like n-gram modelling and the Viterbi al-
gorithm to equip the model with a good language prior. Although the scope
of the problem was originally restricted to well scanned documents without
erasure, we later expand it to handle documents with erasure. Similarly one
can extend the model to handle joined glyphs. To do this, one just needs to
initially generate candidate splits for the edges of the original linear graph
of Figure 15. However, even with this expanded scope, we can not expect
to achieve human-level performance. This is because the language models
 TELUGU OCR FRAMEWORK USING DEEP LEARNING 31

employed by humans are are far more complex (as studied in greater detail
in Natural Language Processing).
Our framework does not generalize to scripts like Devanagari and Ara-
bic. These are written with words as connected components, whereas, our
system relies heavily on being able to segment the printed text. Modern
Deep Learning techniques like Recurrent Neural Networks with Connection-
ist Temporal Classication(CTC) Graves et al. (2009) overcome this design
limitation. They do not need the data to be segmented in a way such that
there is a one-to-one correspondence between input samples and output la-
bels. While CTC broadens the scope of the problem, it more dicult to
train. It might also need a more complicated language model. It would be
interesting to compare a CTC based framework with ours as a reference.

References.
Abu-Mostafa, Y. S. (1990). Learning from hints in neural networks. Journal of com-
plexity 6 192198.
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Berg-
eron, A., Bouchard, N. and Bengio, Y. (2012). Theano: new features and speed
improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Work-
shop.
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G.,
Turian, J., Warde-Farley, D. and Bengio, Y. (2010). Theano: a CPU and GPU
Math Expression Compiler. In Proceedings of the Python for Scientic Computing Con-
ference (SciPy).
Bishop, C. M. (1995). Training with noise is equivalent to Tikhonov regularization. Neural
computation 7 108116.
Bloomberg, D. (2007). Leptonica: An open source C library for ecient image processing,
analysis and operation.
Cirean, D. and Schmidhuber, J. (2013). Multi-column deep neural networks for oine
handwritten Chinese character classication. arXiv preprint arXiv:1309.0261.
Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathe-
matics of control, signals and systems 2 303314.
Graves, A., Liwicki, M., Fernndez, S., Bertolami, R., Bunke, H. and Schmidhu-
ber, J. (2009). A novel connectionist system for unconstrained handwriting recognition.
Pattern Analysis and Machine Intelligence, IEEE Transactions on 31 855868.
He, K., Zhang, X., Ren, S. and Sun, J. (2015). Delving deep into rectiers: Surpassing
human-level performance on imagenet classication. arXiv preprint arXiv:1502.01852.
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I. and Salakhutdi-
nov, R. R. (2012). Improving neural networks by preventing co-adaptation of feature
detectors. arXiv preprint arXiv:1207.0580.
Jawahar, C., Kumar, M. P. and Kiran, S. R. (2003). A bilingual OCR for Hindi-Telugu
documents and its applications. In null 408. IEEE.
Krizhevsky, A., Sutskever, I. and Hinton, G. E. (2012). Imagenet classication with
deep convolutional neural networks. In Advances in neural information processing sys-
tems 10971105.
32 R. ACHANTA ET AL.

Kumar, P. P., Bhagvati, C., Negi, A., Agarwal, A. and Deekshatulu, B. L. (2011).
Towards improving the accuracy of Telugu OCR systems. In Document Analysis and
Recognition (ICDAR), 2011 International Conference on 910914. IEEE.
Lakshmi, C. V. and Patvardhan, C. (2002). A multi-font OCR system for printed
Telugu text. In Language Engineering Conference, 2002. Proceedings 717. IEEE.
LeCun, Y., Bottou, L., Bengio, Y. and Haffner, P. (1998). Gradient-based learning
applied to document recognition. Proceedings of the IEEE 86 22782324.
Nair, V. and Hinton, G. E. (2010). Rectied linear units improve restricted boltzmann
machines. In Proceedings of the 27th International Conference on Machine Learning
(ICML-10) 807814.
Negi, A., Bhagvati, C. and Krishna, B. (2001). An OCR system for Telugu. In Docu-
ment Analysis and Recognition, 2001. Proceedings. Sixth International Conference on
11101114. IEEE.
Pujari, A. K., Naidu, C. D., Rao, M. S. and Jinaga, B. (2004). An intelligent character
recognizer for Telugu scripts using multiresolution analysis and associative memory.
Image and Vision Computing 22 12211227.
Rajasekaran, S. and Deekshatulu, B. (1977). Recognition of printed Telugu charac-
ters. Computer graphics and image processing 6 335360.
Rao, P. and Ajitha, T. (1995). Telugu script recognition-a feature based approach.
In Document Analysis and Recognition, 1995., Proceedings of the Third International
Conference on 1 323326. IEEE.
Simard, P. Y., Steinkraus, D. and Platt, J. C. (2003). Best practices for convolu-
tional neural networks applied to visual document analysis. In 2013 12th International
Conference on Document Analysis and Recognition 2 958958. IEEE Computer Society.
Simard, P. Y., LeCun, Y. A., Denker, J. S. and Victorri, B. (1998). Transformation
invariance in pattern recognitiontangent distance and tangent propagation. In Neural
networks: tricks of the trade 239274. Springer.
Sukhaswami, M., Seetharamulu, P. and Pujari, A. K. (1995). Recognition of Telugu
characters using neural networks. International journal of neural systems 6 317357.
Viterbi, A. J. (1967). Error bounds for convolutional codes and an asymptotically opti-
mum decoding algorithm. Information Theory, IEEE Transactions on 13 260269.
Wager, S., Wang, S. and Liang, P. S. (2013). Dropout training as adaptive regulariza-
tion. In Advances in Neural Information Processing Systems 351359.

Address of the authors

Department of Statistics
Stanford University
390, Serra Mall
Stanford, CA, USA
E-mail: rakesha@stanford.edu
hastie@stanford.edu URL: http://stanford.edu/hastie