Abstract
Generative Adversarial Networks, also called GANs, have been able to produce realistic looking handwritten images. However, training a GAN to generate usable data for improving handwriting recognition is a chicken and egg problem. In a low-resource language scenario, it would be beneficial to have a method that can generate more labeled data. But training such a GAN requires an amount of data that would not be available in a low-resource setting.
In this paper, we present our work in data augmentation with a GAN that is independent of language and can be used to generate handwritten images by learning a mapping between printed and handwritten text. Our method is able to leverage training data from a source language and generate handwriting in a different target language. We show that in scenarios with adequate amounts of target language data, similar improvements in WER can be made by augmenting with either synthetic handwritten or printed text. However, in low resource scenarios, our GAN generated handwriting improves recognition results by 5–10% absolute over the baseline and 3–5% absolute over adding rendered printed text.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alonso, E., Moysset, B., Messina, R.: Adversarial generation of handwritten text images conditioned on sequences. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 481–486. IEEE Computer Society, Los Alamitos (2019). https://doi.org/10.1109/ICDAR.2019.00083, https://doi.ieeecomputersociety.org/10.1109/ICDAR.2019.00083
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473 (2014)
Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=B1xsqj09Fm
Carbune, V., et al.: Fast multi-language LSTM-based online handwriting recognition. Int. J. Doc. Anal. Recogn. (IJDAR) 23(2), 89–102 (2020). https://doi.org/10.1007/s10032-020-00350-4
CFT SL: Cyrillic handwriting dataset (2022). https://www.kaggle.com/datasets/constantinwerner/cyrillic-handwriting-dataset
Etter, D., Rawls, S., Carpenter, C., Sell, G.: A synthetic recipe for OCR. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 864–869 (2019). https://doi.org/10.1109/ICDAR.2019.00143
Fogel, S., Averbuch-Elor, H., Cohen, S., Mazor, S., Litman, R.: ScrabbleGan: semi-supervised varying length handwritten text generation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Goodfellow, I., et al.: Generative adversarial networks. In: Advances in Neural Information Processing Systems, vol. 3 (2014). https://doi.org/10.1145/3422622
Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural nets. In: ICML ’06: Proceedings of the International Conference on Machine Learning (2006)
Graves, A., Fernández, S., Schmidhuber, J.: Bidirectional LSTM networks for improved phoneme classification and recognition. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds.) ICANN 2005. LNCS, vol. 3697, pp. 799–804. Springer, Heidelberg (2005). https://doi.org/10.1007/11550907_126
Heil, R., Vats, E., Hast, A.: Strikethrough removal from handwritten words using CycleGANs. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12824, pp. 572–586. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86337-1_38
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf
Huu, M.K.N., Ho, S.T., Nguyen, V.T., Ngo, T.D.: Multilingual-GAN: a multilingual GAN-based approach for handwritten generation. In: 2021 International Conference on Multimedia Analysis and Pattern Recognition (MAPR), pp. 1–6 (2021). https://doi.org/10.1109/MAPR53640.2021.9585285
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Kong, Y., et al.: Look closer to supervise better: One-shot font generation via component-based discriminator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13482–13491 (2022)
Lee, D., Ismael, S., Grimes, S., Doermann, D., Strassel, S., Chen, S.: MADCAT phase 1 training set (2012). https://doi.org/10.35111/9bm5-nz55
Lee, D., Ismael, S., Grimes, S., Doermann, D., Strassel, S., Chen, S.: MADCAT phase 2 training set (2013). https://doi.org/10.35111/044b-ah68
Lee, D., Ismael, S., Grimes, S., Doermann, D., Strassel, S., Chen, S.: MADCAT phase 3 training set (2013). https://doi.org/10.35111/w1px-d922
Li, C., Taniguchi, Y., Lu, M., Konomi, S.: Few-shot font style transfer between different languages. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 433–442 (2021)
Liu, W., Liu, F., Ding, F., He, Q., Yi, Z.: XMP-FONT: self-supervised cross-modality pre-training for few-shot font generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7905–7914 (2022)
Luo, C., Zhu, Y., Jin, L., Li, Z., Peng, D.: SLOGAN: handwriting style synthesis for arbitrary-length and out-of-vocabulary text. IEEE Trans. Neural Netw. Learn. Syst. (2022)
Marti, U.V., Bunke, H.: The IAM-database: an English sentence database for offline handwriting recognition. Int. J. Doc. Anal. Recogn. 5, 39–46 (2002)
Nayef, N., et al.: ICDAR 2019 robust reading challenge on multi-lingual scene text detection and recognition - RRC-MLT-2019 (2019). https://doi.org/10.48550/ARXIV.1907.00945, https://arxiv.org/abs/1907.00945
Nguyen, H.T., Nguyen, C.T., Nakagawa, M.: ICFHR 2018 - competition on vietnamese online handwritten text recognition using HANDS-VNonDB (VOHTR2018). In: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 494–499 (2018). https://doi.org/10.1109/ICFHR-2018.2018.00092
Park, S., Chun, S., Cha, J., Lee, B., Shim, H.: Multiple heads are better than one: few-shot font generation with multiple localized experts. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13900–13909 (2021)
Seitzer, M.: PyTorch-FID: FID Score for PyTorch. https://github.com/mseitzer/pytorch-fid (2020). version 0.2.1
Sharma, M., Verma, A., Vig, L.: Learning to clean: a GAN perspective. In: Carneiro, G., You, S. (eds.) ACCV 2018. LNCS, vol. 11367, pp. 174–185. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21074-8_14
Tang, L., et al.: Few-shot font generation by learning fine-grained local styles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7895–7904 (2022)
Veit, A., Matera, T., Neumann, L., Matas, J., Belongie, S.: Coco-text: dataset and benchmark for text detection and recognition in natural images. arXiv preprint: arXiv:1601.07140 (2016). http://vision.cornell.edu/se3/wp-content/uploads/2016/01/1601.07140v1.pdf
Wang, Y., et al.: Espresso: a fast end-to-end neural speech recognition toolkit. In: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 136–143 (2019). https://doi.org/10.1109/ASRU46091.2019.9003968
Wei, H., Liu, K., Zhang, J., Fan, D.: Data augmentation based on CycleGAN for improving woodblock-printing Mongolian words recognition. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12824, pp. 526–537. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86337-1_35
Wen, C., et al.: Handwritten Chinese font generation with collaborative stroke refinement. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3882–3891 (2021)
Yim, M., Kim, Y., Cho, H.-C., Park, S.: SynthTIGER: synthetic text image GEneratoR towards better text recognition models. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12824, pp. 109–124. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86337-1_8
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Computer Vision (ICCV), 2017 IEEE International Conference on (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chang, C.C., Perera, L.P.G., Khudanpur, S. (2023). Crosslingual Handwritten Text Generation Using GANs. In: Coustaty, M., Fornés, A. (eds) Document Analysis and Recognition – ICDAR 2023 Workshops. ICDAR 2023. Lecture Notes in Computer Science, vol 14194. Springer, Cham. https://doi.org/10.1007/978-3-031-41501-2_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-41501-2_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41500-5
Online ISBN: 978-3-031-41501-2
eBook Packages: Computer ScienceComputer Science (R0)
