Enriching Image Archives via Facial Recognition
Authors: 
Kenzo Milleville, Alec Van den Broeck, Nastasia Vanderperren, Rony Vissers, Matthias Priem, Nico Van de Weghe, Steven VerstocktAuthors Info & Claims
Kenzo Milleville, Alec Van den Broeck, Nastasia Vanderperren, Rony Vissers, Matthias Priem, Nico Van de Weghe, Steven VerstocktAuthors Info & ClaimsArticle No.: 78, Pages 1 - 18
Abstract
The digitization of image archives across the globe has opened up vast collections of libraries, museums, and cultural heritage institutions. These collections provide valuable historical information to the public and researchers. Many image collections have little metadata describing who or what is depicted in a structured format, making it difficult to search for specific persons. This work presents a facial recognition pipeline to enrich these collections by recognizing the persons in each image. A reference dataset of over 6,000 known persons was constructed and facial recognition was performed on a dataset of over 150 thousand images. Detected faces were matched with the known faces using a similarity score on the face embeddings. We developed an interactive labeling tool to efficiently validate the face recognition predictions. A total of 182 thousand detected faces were labeled with this tool. Using a minimum similarity score of 0.5, the face recognition model achieved a precision of 0.936 and identified over 62 thousand persons from the image archives. We show how clustering can be used to identify new persons that were not included in the reference dataset. Furthermore, we highlight the potential of facial recognition to enhance the accessibility of the collections and offer new insights.
References
[1]
2022. InsightFace: State of the art deep face analysis library. (2022). Retrieved December 29, 2022 from https://github.com/deepinsight/insightface
[2]
Fadi Boutros, Naser Damer, Florian Kirchbuchner, and Arjan Kuijper. 2022. Elasticface: Elastic margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1578–1587.
[3]
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research). Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, 77–91. Retrieved from https://proceedings.mlr.press/v81/buolamwini18a.html
[4]
Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman. 2018. Vggface2: A dataset for recognising faces across pose and age. In Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG’18). IEEE, 67–74.
[5]
Brandon Castellano. 2022. PySceneDetect, Intelligent scene cut detection and video splitting tool. (2022). Retrieved December 29, 2022 from https://scenedetect.com/en/latest/
[6]
Tim Causer and Melissa Terras. 2014. Crowdsourcing bentham: Beyond the traditional boundaries of academic history. International Journal of Humanities and Arts Computing 8, 1 (2014), 46–64.
[7]
Giovanni Colavizza, Tobias Blanke, Charles Jeurgens, and Julia Noordegraaf. 2021. Archives and AI: An overview of current debates and future perspectives. Journal on Computing and Cultural Heritage 15, 1, Article 4 (Dec. 2021), 15 pages. DOI:
[8]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248–255.
[9]
Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5203–5212.
[10]
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4690–4699.
[11]
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining. Vol. 96. 226–231.
[12]
Jia Guo, Jiankang Deng, Alexandros Lattas, and Stefanos Zafeiriou. 2022. Sample and computation redistribution for efficient face detection. In International Conference on Learning Representations. https://openreview.net/forum?id=RhB1AdoFfGE
[13]
Gary B. Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In Proceedings of the Workshop on Faces in‘Real-Life’Images: Detection, Alignment, and Recognition.
[14]
Manjeeta R. Kale, Priti P. Rege, and Radhika D. Joshi. 2023. Designing a dual-level facial expression evaluation system for performers using geometric features and petri nets. Journal on Computing and Cultural Heritage (Feb. 2023). DOI:Just Accepted.
[15]
Jian Li, Yabiao Wang, Changan Wang, Ying Tai, Jianjun Qian, Jian Yang, Chengjie Wang, Jilin Li, and Feiyue Huang. 2019. DSFD: Dual Shot Face Detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5060–5069.
[16]
Pasquale Lisena, Jorma Laaksonen, and Raphaël Troncy. 2021. FaceRec: An interactive framework for face recognition in video archives. In Proceedings of the DataTV 2021, 2nd International Workshop on Data-Driven Personalisation of Television.
[17]
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. 2019. Mediapipe: A framework for building perception pipelines. arXiv:1906.08172. Retrieved from https://arxiv.org/abs/1906.08172
[18]
Kenzo Milleville, Alec Van den Broeck, Rony Vissers, Bart Magnus, Nastasia Vanderperren, Astrid Vergauwe, Ellen Van Keer, Tom Ruette, and Steven Verstockt. 2022. FAME video browser—face recognition based metadata generation for performing art videos. In Proceedings of the DH Benelux 2022-ReMIX: Creation and Alteration in DH (hybrid). Zenodo, 1–4. DOI:
[19]
Vikram Mohanty, David Thames, Sneha Mehta, and Kurt Luther. 2020. Photo sleuth: Identifying historical portraits with face recognition and crowdsourced human expertise. ACM Transactions on Interactive Intelligent Systems (TiiS) 10, 4 (2020), 1–36.
[20]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8748–8763. https://proceedings.mlr.press/v139/radford21a.html
[21]
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 815–823.
[22]
Benoit Seguin. 2018. The replica project: Building a visual search engine for art historians. XRDS: Crossroads, The ACM Magazine for Students 24, 3 (2018), 24–29.
[23]
Benoît Laurent Auguste Seguin. 2018. Making large art historical photo archives searchable. (2018), 169. DOI:
[24]
Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. 2014. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1701–1708.
[25]
Philipp Terhörst, Jan Niklas Kolf, Marco Huber, Florian Kirchbuchner, Naser Damer, Aythami Morales Moreno, Julian Fierrez, and Arjan Kuijper. 2022. A comprehensive study on face recognition biases beyond demographics. IEEE Transactions on Technology and Society 3, 1 (2022), 16–30. DOI:
[26]
Steven Verstockt, Kenzo Milleville, Dilawar Ali, Francisco Porras-Bernandez, Georg Gartner, and Nico Van de Weghe. 2019. EURECA: European region enrichment in city archives and collections. In Proceedings of the 14th ICA conference: Digital Approaches to Cartographic Heritage. Aristoteleio Panepistimio Thessalonikis (APTh), 161–169.
[27]
Paul Viola and Michael Jones. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Vol. 1, IEEE, I–I.
[28]
Mei Wang and Weihong Deng. 2018. Deep face recognition: A survey. Neurocomputing 429 (2021), 215–244.
[29]
Melvin Wevers and Thomas Smits. 2020. Detecting faces, visual medium types, and gender in historical advertisements, 1950–1995. In Proceedings of the Computer Vision—ECCV 2020 Workshops. Adrien Bartoli and Andrea Fusiello (Eds.), Springer International Publishing, Cham, 77–91.
[30]
Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li. 2014. Learning face representation from scratch. arXiv Preprint arXiv:1411.7923 (2014).
[31]
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23, 10 (2016), 1499–1503.
[32]
Yuxiang Zhao and Qinghua Zhu. 2014. Evaluation on crowdsourcing research: Current status and future direction. Information Systems Frontiers 16, 3 (2014), 417–434.
[33]
Zheng Zhu, Guan Huang, Jiankang Deng, Yun Ye, Junjie Huang, Xinze Chen, Jiagang Zhu, Tian Yang, Jiwen Lu, Dalong Du, and Jie Zhou. 2021. WebFace260M: A benchmark unveiling the power of million-scale deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10492–10502.
Index Terms
- Enriching Image Archives via Facial Recognition
Recommendations
Facial expression recognition with Convolutional Neural Networks
Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine ...
Recognizing Action Units for Facial Expression Analysis
Most automatic expression analysis systems attempt to recognize a small set of prototypic expressions, such as happiness, anger, surprise, and fear. Such prototypic expressions, however, occur rather infrequently. Human emotions and intentions are more ...
Joint facial expression recognition and intensity estimation based on weighted votes of image sequences
A framework for joint facial expression recognition and intensity estimation from image sequences is proposed.A feature representation based on weighted votes is also proposed.Superior performance in estimating facial expression intensities.Promising ...
Comments
Information & Contributors
Information
Published In
December 2023
473 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 16 November 2023
Online AM: 05 July 2023
Accepted: 18 May 2023
Revised: 14 April 2023
Received: 30 December 2022
Published in JOCCH Volume 16, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Ghent University, Imec, Meemoo, the Belgian Science Policy Office (BELSPO)
- Flanders Department of Culture, Youth, and Media
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 631Total Downloads
- Downloads (Last 12 months)429
- Downloads (Last 6 weeks)46
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in