research-article

Handwritten Text Recognition from Crowdsourced Annotations

Authors:

Solène Tarride,

Mélodie Boillet,

Harold Mouchère,

Christopher KermorvantAuthors Info & Claims

HIP '23: Proceedings of the 7th International Workshop on Historical Document Imaging and Processing

Pages 1 - 6

https://doi.org/10.1145/3604951.3605517

Published: 25 August 2023 Publication History

Abstract

In this paper, we explore different ways of training a model for handwritten text recognition when multiple imperfect or noisy transcriptions are available. We consider various training configurations, such as selecting a single transcription, retaining all transcriptions, or computing an aggregated transcription from all available annotations. In addition, we evaluate the impact of quality-based data selection, where samples with low agreement are removed from the training set. Our experiments are carried out on municipal registers of the city of Belfort (France) written between 1790 and 1946. The results show that computing a consensus transcription or training on multiple transcriptions are good alternatives. However, selecting training samples based on the degree of agreement between annotators introduces a bias in the training data and does not improve the results. Our dataset is publicly available on Zenodo.

References

[1]

Samantha Blickhan, Coleman Krawczyk, Daniel Hanson, Amy Boyer, Andrea Simenstad, and Victoria van Hyning. 2019. Individual vs. Collaborative Methods of Crowdsourced Transcription. In Journal of Data Mining and Digital Humanities. https://doi.org/10.46298/jdmdh.5759

[2]

Mélodie Boillet, Christopher Kermorvant, and Thierry Paquet. 2021. Multiple Document Datasets Pre-training Improves Text Line Detection With Deep Neural Networks. In 25th International Conference on Pattern Recognition. 2134–2141. https://doi.org/10.1109/ICPR48806.2021.9412447

[3]

Denis Coquenet, Clement Chatelain, and Thierry Paquet. 2023. DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023), 1–17. https://doi.org/10.1109/tpami.2023.3235826

Digital Library

[4]

Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality Control in Crowdsourcing. In ACM Computing Surveys. 1–40. https://doi.org/10.1145/3148148

Digital Library

[5]

Wafaa S. El-Kassas, Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed. 2021. Automatic text summarization: A comprehensive survey. In Expert Systems with Applications. 113679. https://doi.org/10.1016/j.eswa.2020.113679

[6]

J.G. Fiscus. 1997. A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction (ROVER). In IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings. 347–354. https://doi.org/10.1109/ASRU.1997.659110

[7]

Zhengqi Gao, Fan-Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Marc Engeler, Antonio Burazer, Linda Wildling, Luca Daniel, and Duane S. Boning. 2022. Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusion. In Computer Vision – ECCV 2022. 407–422. https://doi.org/10.1007/978-3-031-20053-3_24

Digital Library

[8]

The Interedition Development Group. 2010-2019. https://collatex.net/ Collatex software home page. Accessed: 2023-04-14.

[9]

Jiyi Li. 2020. Crowdsourced Text Sequence Aggregation Based on Hybrid Reliability and Representation. In 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1761–1764. https://doi.org/10.1145/3397271.3401239

Digital Library

[10]

Jiyi Li and Fumiyo Fukumoto. 2019. A Dataset of Crowdsourced Word Sequences: Collections and Answer Aggregation for Ground Truth Creation. In Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. 24–28. https://doi.org/10.18653/v1/D19-5904

[11]

Joan Puigcerver. 2017. Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition?. In 14th International Conference on Document Analysis and Recognition. 67–72. https://doi.org/10.1109/ICDAR.2017.20

[12]

Filipe Rodrigues and Francisco Pereira. 2018. Deep learning from crowds. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. 1611–1618. https://doi.org/10.5555/3504035.3504232

Digital Library

[13]

Dmitry Ustalov, Nikita Pavlichenko, and Boris Tseitlin. 2023. Learning from Crowds with Crowd-Kit. https://arxiv.org/abs/2109.08584

[14]

Alan J. Wecker, Vered Raziel-Kretzmer, Benjamin Kiessling, Daniel Stökl Ben Ezra, Moshe Lavee, Tsvi Kuflik, Dror Elovits, Moshe Schorr, Uri Schor, and Pawel Jablonski. 2022. Tikkoun Sofrim: Making Ancient Manuscripts Digitally Accessible: The Case of Midrash Tanhuma. J. Comput. Cult. Herit. 15, 2, Article 20 (apr 2022), 20 pages. https://doi.org/10.1145/3476776

Digital Library

[15]

Hongxin Wei, Renchunzi Xie, Lei Feng, Bo Han, and Bo An. 2022. Deep Learning From Multiple Noisy Annotators as A Union. In IEEE transactions on neural networks and learning systems.

[16]

Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. 2021. Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. ArXiv abs/2110.12088 (2021).

[17]

Jiaheng Wei, Zhaowei Zhu, Tianyi Luo, Ehsan Amid, Abhishek Kumar, and Yang Liu. 2022. To Aggregate or Not? Learning with Separate Noisy Labels. In CSW@WSDM.

[18]

Zhaowei Zhu, Zihao Dong, and Yang Liu. 2021. Detecting Corrupted Labels Without Training a Model to Predict. In International Conference on Machine Learning.

Cited By

N ASrija BJabeen MKankar KLakshmi BNegi A(2024)A Review on Automated Annotation System for Document Text Images2024 1st International Conference on Cognitive, Green and Ubiquitous Computing (IC-CGU)10.1109/IC-CGU58078.2024.10530793(1-6)Online publication date: 1-Mar-2024
https://doi.org/10.1109/IC-CGU58078.2024.10530793
Kermorvant CBardou EBlanco MAbadie B(2024)Callico: A Versatile Open-Source Document Image Annotation PlatformDocument Analysis and Recognition - ICDAR 202410.1007/978-3-031-70543-4_20(338-353)Online publication date: 9-Sep-2024
https://doi.org/10.1007/978-3-031-70543-4_20
Mondal ATulsyan KJawahar C(2024)Bridging the Gap in Resource for Offline English Handwritten Text RecognitionDocument Analysis and Recognition - ICDAR 202410.1007/978-3-031-70536-6_25(413-428)Online publication date: 3-Sep-2024
https://doi.org/10.1007/978-3-031-70536-6_25

Index Terms

Handwritten Text Recognition from Crowdsourced Annotations

Recommendations

iiit-indic-hw-words: A Dataset for Indic Handwritten Text Recognition
Document Analysis and Recognition – ICDAR 2021
Abstract
Handwritten text recognition (htr) for Indian languages is not yet a well-studied problem. This is primarily due to the unavailability of large annotated datasets in the associated scripts. Existing datasets are small in size. They also use small ...
Learning to Read L’Infinito: Handwritten Text Recognition with Synthetic Training Data
Computer Analysis of Images and Patterns
Abstract
Deep learning-based approaches to Handwritten Text Recognition (HTR) have shown remarkable results on publicly available large datasets, both modern and historical. However, it is often the case that historical manuscripts are preserved in small ...
Handwritten text recognition for historical documents in the transcriptorium project
DATeCH '14: Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage

Transcription of historical handwritten documents is a crucial problem for making easier the access to these documents to the general public. Currently, huge amount of historical handwritten documents are being made available by on-line portals ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

HIP '23: Proceedings of the 7th International Workshop on Historical Document Imaging and Processing

August 2023

117 pages

ISBN:9798400708411

DOI:10.1145/3604951

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

HIP '23

HIP '23: 7th International Workshop on Historical Document Imaging and Processing

August 25 - 26, 2023

CA, San Jose, USA

Acceptance Rates

Overall Acceptance Rate 52 of 90 submissions, 58%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
47
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)3

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

N ASrija BJabeen MKankar KLakshmi BNegi A(2024)A Review on Automated Annotation System for Document Text Images2024 1st International Conference on Cognitive, Green and Ubiquitous Computing (IC-CGU)10.1109/IC-CGU58078.2024.10530793(1-6)Online publication date: 1-Mar-2024
https://doi.org/10.1109/IC-CGU58078.2024.10530793
Kermorvant CBardou EBlanco MAbadie B(2024)Callico: A Versatile Open-Source Document Image Annotation PlatformDocument Analysis and Recognition - ICDAR 202410.1007/978-3-031-70543-4_20(338-353)Online publication date: 9-Sep-2024
https://doi.org/10.1007/978-3-031-70543-4_20
Mondal ATulsyan KJawahar C(2024)Bridging the Gap in Resource for Offline English Handwritten Text RecognitionDocument Analysis and Recognition - ICDAR 202410.1007/978-3-031-70536-6_25(413-428)Online publication date: 3-Sep-2024
https://doi.org/10.1007/978-3-031-70536-6_25
Jang HChoi HKim BKim SKoo G(2023)Two-Stage Billet Identification Number Recognition Using Label DistributionIEEE Access10.1109/ACCESS.2023.333390411(129311-129319)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3333904

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents