research-article

Fusion of Spatio-temporal Information for Indic Word Recognition Combining Online and Offline Text Data

Authors:

Subham Mukherjee,

Partha Pratim RoyAuthors Info & Claims

ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), Volume 19, Issue 2

Article No.: 33, Pages 1 - 24

https://doi.org/10.1145/3364533

Published: 21 November 2019 Publication History

Abstract

We present a novel Indic handwritten word recognition scheme by fusion of spatio-temporal information extracted from handwritten images. The main challenge in Indic word recognition lies in its complexity because of modifiers, touching characters, and compound characters. Hidden Markov Models (HMMs) are being used to model such data due to their ability to learn sequential data, however, the recognition performance is not satisfactory. We propose here a Long Short-Term Memory (LSTM)-based architecture for offline Indic word recognition. Offline recognition methods usually involve spatial data, whereas it has been observed that online recognition schemes show better performance than the offline methodologies. Online information usually refers to the temporal information obtained from the strokes of the pen tip while writing, which is missing in offline word images. In this article, an effort has been made to extract the online temporal information from offline images using stroke recovery and later it is combined with spatial information in LSTM architecture. During recognition, the character models are trained using both offline and extracted pseudo-online handwritten data separately. Finally, a novel fusion scheme has been used to combine them together. From the experiment, it is noted that recognition performance of handwritten Indic words improves considerably due to the fusion scheme of spatial and temporal data.

References

[1]

Mohamed Nidhal Abdi and Maher Khemakhem. 2015. A model-based approach to offline text-independent Arabic writer identification and verification. Pattern Recog. 48, 5 (2015), 1890--1903.

Digital Library

[2]

Chandranath Adak, Bidyut B. Chaudhuri, and Michael Blumenstein. 2016. Offline cursive Bengali word recognition using CNNs with a recurrent model. In Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition (ICFHR’16). IEEE, 429--434.

[3]

Irfan Ahmad and Sabri A. Mahmoud. 2012. Arabic bank check analysis and zone extraction. In Proceedings of the International Conference on Image Analysis and Recognition. Springer, 141--148.

[4]

Jawad H. AlKhateeb, Jinchang Ren, Jianmin Jiang, and Husni Al-Muhtaseb. 2011. Offline handwritten Arabic cursive text recognition using hidden Markov models and re-ranking. Pattern Recog. Lett. 32, 8 (2011), 1081--1088.

Digital Library

[5]

Sandhya Arora, Debotosh Bhattacharjee, Mita Nasipuri, Dipak Kumar Basu, and Mahantapas Kundu. 2010. Multiple classifier combination for off-line handwritten Devnagari character recognition. Retrieved from Arxiv Preprint Arxiv:1006.5913 (2010).

[6]

Ahmad-Montaser Awal, Harold Mouchère, and Christian Viard-Gaudin. 2014. A global learning approach for an online handwritten mathematical expression recognition system. Pattern Recog. Lett. 35 (2014), 68--77.

Digital Library

[7]

A. Bharath and Sriganesh Madhvanath. 2012. HMM-based lexicon-driven and lexicon-free word recognition for online handwritten Indic scripts. IEEE Trans. Pattern Anal. Mach. Intell. 34, 4 (2012), 670--682.

Digital Library

[8]

Nilanjana Bhattacharya and Umapada Pal. 2012. Stroke segmentation and recognition from bangla online handwritten text. In Proceedings of the International Conference on Frontiers in Handwriting Recognition. 740--745.

Digital Library

[9]

Fadi Biadsy, Jihad El-Sana, and Nizar Habash. 2006. Online Arabic handwriting recognition using hidden Markov models. In Proceedings of the 10th International Workshop on Frontiers in Handwriting Recognition. Suvisoft.

[10]

Anne-Laure Bianne-Bernard, Farès Menasri, Ramy Al-Hajj Mohamad, Chafic Mokbel, Christopher Kermorvant, and Laurence Likforman-Sulem. 2011. Dynamic and contextual information in HMM modeling for handwritten word recognition. IEEE Trans. Pattern Anal. Mach. Intell. 33, 10 (2011), 2066--2080.

Digital Library

[11]

Horst Bunke and Tamás Varga. 2007. Off-line Roman cursive handwriting recognition. In Digital Document Processing. Springer, 165--183.

[12]

Youssouf Chherawala, Partha Pratim Roy, and Mohamed Cheriet. 2016. Feature set evaluation for offline handwriting recognition systems: Application to the recurrent neural network model. IEEE Trans. Cyber. 46, 12 (2016), 2825--2836.

[13]

Youssouf Chherawala, Partha Pratim Roy, and Mohamed Cheriet. 2017. Combination of context-dependent bidirectional long short-term memory classifiers for robust offline handwriting recognition. Pattern Recog. Lett. 90 (2017), 58--64.

Digital Library

[14]

P. Doetsch, M. Kozielski, and H. Ney. 2014. Fast and robust training of recurrent neural networks for offline handwriting recognition. In Proceedings of the 14th International Conference on Frontiers in Handwriting Recognition. 279--284.

[15]

Kartik Dutta, Praveen Krishnan, Minesh Mathew, and C. V. Jawahar. 2017. Towards accurate handwritten word recognition for Hindi and Bangla. In Proceedings of the National Conference on Computer Vision, Pattern Recognition, Image Processing, and Graphics. Springer, 470--480.

[16]

S. Espana-Boquera, M. J. Castro-Bleda, J. Gorbe-Moya, and F. Zamora-Martinez. 2011. Improving offline handwritten text recognition with hybrid HMM/ANN models. IEEE Trans. Pattern Anal. Mach. Intell. 33, 4 (2011), 767--779.

Digital Library

[17]

Vahid Ghods, Ehsanollah Kabir, and Farbod Razzazi. 2013. Decision fusion of horizontal and vertical trajectories for recognition of online Farsi subwords. Eng. Appl. Artific. Intell. 26, 1 (2013), 544--550.

Digital Library

[18]

Rajib Ghosh, Pradeep Kumar, and Partha Pratim Roy. 2019. A Dempster-Shafer theory based classifier combination for online signature recognition and verification systems. Int. J. Mach. Learn. Cyber. 10, 9 (2019), 2467--2482.

[19]

Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning. ACM, 369--376.

Digital Library

[20]

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. 2001. Gradient flow in recurrent nets: The difficulty of learning long-term dependencies.

[21]

Yoshiharu Kato and Makoto Yasuhara. 2000. Recovery of drawing order from single-stroke handwriting images. IEEE Trans. Pattern Anal. Mach. Intell. 22, 9 (2000), 938--949.

Digital Library

[22]

JinHyung Kim and Bong-Kee Sin. 2014. Online handwriting recognition. In Handbook of Document Image Processing and Recognition. Springer, 887--915.

[23]

Pradeep Kumar, Himaanshu Gauba, Partha Pratim Roy, and Debi Prosad Dogra. 2017a. Coupled HMM-based multi-sensor data fusion for sign language recognition. Pattern Recog. Lett. 86 (2017), 1--8.

Digital Library

[24]

Pradeep Kumar, Himaanshu Gauba, Partha Pratim Roy, and Debi Prosad Dogra. 2017b. A multimodal framework for sensor based sign language recognition. Neurocomputing 259 (2017), 21--38.

[25]

Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, and Debi Prosad Dogra. 2016. Study of text segmentation and recognition using leap motion sensor. IEEE Sens. J. 17, 5 (2016), 1293--1301.

[26]

Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, and Debi Prosad Dogra. 2017c. 3D text segmentation and recognition using leap motion. Multimedia Tools Appl. 76, 15 (2017), 16491--16510.

Digital Library

[27]

Pradeep Kumar, Rajkumar Saini, Partha Pratim Roy, and Debi Prosad Dogra. 2018. A position and rotation invariant framework for sign language recognition (SLR) using Kinect. Multimedia Tools Appl. 77, 7 (2018), 8823--8846.

Digital Library

[28]

Longin Jan Latecki, Quan-nan Li, Xiang Bai, and Wen-yu Liu. 2007. Skeletonization using SSM of the distance transform. In Proceedings of the International Conference on Image Processing, Vol. 5. V--349.

[29]

Umapada Pal, Ramachandran Jayadevan, and Nabin Sharma. 2012a. Handwriting recognition in Indian regional scripts: A survey of offline techniques. ACM Trans. Asian Lang. Inform. Proc. 11, 1 (2012), 1.

Digital Library

[30]

Umapada Pal, Ramachandran Jayadevan, and Nabin Sharma. 2012b. Handwriting recognition in Indian regional scripts: A survey of offline techniques. ACM Trans. Asian Lang. Inform. Proc. 11, 1 (2012), 1.

Digital Library

[31]

Mohammad Tanvir Parvez and Sabri A. Mahmoud. 2013a. Arabic handwriting recognition using structural and syntactic pattern attributes. Pattern Recog. 46, 1 (2013), 141--154.

Digital Library

[32]

Mohammad Tanvir Parvez and Sabri A. Mahmoud. 2013b. Offline Arabic handwritten text recognition: A survey. Comput. Surv. 45, 2 (2013), 23.

Digital Library

[33]

Réjean Plamondon and Claudio M. Privitera. 1999. The segmentation of cursive handwriting: An approach based on off-line recovery of the motor-temporal information. IEEE Trans. Image Proc. 8, 1 (1999), 80--91.

Digital Library

[34]

Steve Procter and John Illingworth. 1998. Combining HMM classifiers in a handwritten text recognition system. In Proceedings of the International Conference on Image Processing, Vol. 2. 934--938.

[35]

Yu Qiao, Jianzhuang Liu, and Xiaoou Tang. 2007. Offline signature verification using online handwriting registration. In Proceedings of the Conference on Computer Vision and Pattern Recognition. 1--8.

[36]

Kaushik Roy, Szilárd Vajda, Umapada Pal, Bidyut Baran Chaudhuri, and Abdel Belaïd. 2005. A system for Indian postal automation. In Proceedings of the 8th International Conference on Document Analysis and Recognition. 1060--1064.

Digital Library

[37]

Partha Pratim Roy, Ayan Kumar Bhunia, Ayan Das, Prasenjit Dey, and Umapada Pal. 2016. HMM-based Indic handwritten word recognition using zone segmentation. Pattern Recog. 60 (Dec. 2016), 1057--1075.

[38]

Partha Pratim Roy, Ayan Kumar Bhunia, Ayan Das, Prithviraj Dhar, and Umapada Pal. 2017b. Keyword spotting in doctor’s handwriting on medical prescriptions. Expert Syst. Appl. 76 (2017), 113--128.

Digital Library

[39]

Partha Pratim Roy, Ayan Kumar Bhunia, and Umapada Pal. 2017a. HMM-based writer identification in music score documents without staff-line removal. Expert Syst. Appl. 89 (2017), 222--240.

Digital Library

[40]

Partha Pratim Roy, Umapada Pal, and Josep Lladós. 2011. Document seal detection using GHT and character proximity graphs. Pattern Recog. 44, 6 (2011), 1282--1295.

Digital Library

[41]

Partha Pratim Roy, Guoqiang Zhong, and Mohamed Cheriet. 2017c. Tandem hidden Markov models using deep belief networks for offline handwriting recognition. Frontiers of IT 8 EE 18, 7 (2017), 978--988.

[42]

Jaakko Sauvola and Matti Pietikäinen. 2000. Adaptive document image binarization. Pattern Recog. 33, 2 (2000), 225--236.

[43]

Ching Y. Suen and Louisa Lam. 2000. Multiple classifier combination methodologies for different output levels. In Proceedings of the International Workshop on Multiple Classifier Systems. Springer, 52--66.

[44]

Stanford University. 1999a. Neural Networks and Regularization. Retrieved from http://cs231n.github.io/neural-networks-2/#reg.

[45]

Stanford University. 1999b. Unsupervised Learning. Retrieved from http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders.

[46]

Lei Xu, Adam Krzyzak, and Ching Y. Suen. 1992. Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Trans. Syst., Man, Cyber. 22, 3 (1992), 418--435.

[47]

Xu-Yao Zhang, Yoshua Bengio, and Cheng-Lin Liu. 2017. Online and offline handwritten Chinese character recognition: A comprehensive study and new benchmark. Pattern Recog. 61 (2017), 348--360.

Cited By

Jyoti Solanki KDalal SSudhir Sharma SDevi N(2024)Advancements in Offline Handwriting-Based Language Recognition: A Comprehensive Review for Indic Scripts2024 11th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO)10.1109/ICRITO61523.2024.10522174(1-7)Online publication date: 14-Mar-2024
https://doi.org/10.1109/ICRITO61523.2024.10522174
Guo HDong NZhao JLiu Y(2022)Handwritten New Tai Lue Character Recognition Using Convolutional Prior Features and Deep Variationally Sparse Gaussian Process ModelingACM Transactions on Asian and Low-Resource Language Information Processing10.1145/350670021:4(1-25)Online publication date: 20-Jan-2022
https://dl.acm.org/doi/10.1145/3506700

Index Terms

Fusion of Spatio-temporal Information for Indic Word Recognition Combining Online and Offline Text Data
1. Applied computing
  1. Document management and text processing
    1. Document capture
      1. Online handwriting recognition

Recommendations

Character and numeral recognition for non-Indic and Indic scripts: a survey
Abstract
A collection of different scripts is employed in writing languages throughout the world. Character and numeral recognition of a particular script is a key area in the field of pattern recognition. In this paper, we have presented a comprehensive ...
Offline recognition of handwritten Indic scripts: A state-of-the-art survey and future perspectives
Abstract
The handwritten script recognition is one of the most interesting and challenging areas of pattern recognition due to numerous variations in writing styles. Extensive in-depth research work is reported on the recognition of handwritten ...
Highlights
- A comprehensive survey focusing on the recognition of handwritten Indic scripts.
HMM-based Indic handwritten word recognition using zone segmentation

This paper presents a novel approach towards Indic handwritten word recognition using zone-wise information. Because of complex nature due to compound characters, modifiers, overlapping and touching, etc., character segmentation and recognition is a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Asian and Low-Resource Language Information Processing

ACM Transactions on Asian and Low-Resource Language Information Processing Volume 19, Issue 2

March 2020

301 pages

ISSN:2375-4699

EISSN:2375-4702

DOI:10.1145/3358605

Editor:
Imed Zitouni
Microsoft, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 November 2019

Accepted: 01 September 2019

Revised: 01 July 2019

Received: 01 November 2017

Published in TALLIP Volume 19, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
192
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jyoti Solanki KDalal SSudhir Sharma SDevi N(2024)Advancements in Offline Handwriting-Based Language Recognition: A Comprehensive Review for Indic Scripts2024 11th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO)10.1109/ICRITO61523.2024.10522174(1-7)Online publication date: 14-Mar-2024
https://doi.org/10.1109/ICRITO61523.2024.10522174
Guo HDong NZhao JLiu Y(2022)Handwritten New Tai Lue Character Recognition Using Convolutional Prior Features and Deep Variationally Sparse Gaussian Process ModelingACM Transactions on Asian and Low-Resource Language Information Processing10.1145/350670021:4(1-25)Online publication date: 20-Jan-2022
https://dl.acm.org/doi/10.1145/3506700

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents