Facial Image-Based Automatic Assessment of Equine Pain
Pages 2064 - 2076
Abstract
Recognition of pain in animals is essential for their welfare. However, since there is no verbal communication, this assessment depends solely on the ability of the observer to locate visible or audible signs of pain. The use of grimace scales is proven to be efficient in detecting the pain visually, but the assessment quality depends on the level of training of the assessor and the validity is not easily ensured. There is a clear need for automating the pain assessment process. This work provides a system for pain prediction in horses, based on grimace scales. The pipeline automatically determines the quantitative pose of the equine head and finds facial landmarks before classification, proposing a novel scale-normalisation approach for equine heads. The pain estimation is achieved for each facial region of interest separately, following the clinical pain estimation procedure. We introduce a database of horse images, annotated by professional veterinarians for training and assessment. We also propose a data augmentation method to alleviate the data scarcity issues, which relies on generating realistic 3D equine face models based on 2D annotated images. We show that the data augmentation method improves the performance of both quantitative pose estimation and landmark detection. Our results establish a strong baseline for automatic equine pain estimation.
References
[1]
J. Alabort-i-Medina, E. Antonakos, J. Booth, P. Snape, and S. Zafeiriou, “Menpo: A comprehensive platform for parametric image alignment and visual deformable models,” in Proc. 22nd ACM Int. Conf. Multimedia, 2014, pp. 679–682.
[2]
P. H. Andersen et al., “Towards machine recognition of facial expressions of pain in horses,” Animals, vol. 11, no. 6, 2021, Art. no.
[3]
N. Andresen et al., “Towards a fully automated surveillance of well-being status in laboratory mice using deep learning: Starting with facial expression analysis,” PLoS One, vol. 15, no. 4, 2020, Art. no.
[4]
F. Ashley, A. Waterman-Pearson, and H. Whay, “Behavioural assessment of pain in horses and donkeys: Application to clinical practice and future studies,” Equine Vet. J., vol. 37, no. 6, pp. 565–575, 2005.
[5]
K. Ask, M. Rhodin, L.-M. Tamminen, E. Hernlund, and P. Haubro Andersen, “Identification of body behaviors and facial expressions associated with induced orthopedic pain in four equine pain scales,” Animals, vol. 10, no. 11, 2020, Art. no.
[6]
G. Bargshady, X. Zhou, R. C. Deo, J. Soar, F. Whittaker, and H. Wang, “Enhanced deep learning algorithm development to detect pain intensity from facial expression images,” Expert Syst. Appl., vol. 149, 2020, Art. no.
[7]
M. Bartlett et al., “Data mining spontaneous facial behavior with automatic expression coding,” in Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction. Berlin, Germany: Springer, 2008, pp. 1–20.
[8]
F. L. Bookstein, “Principal warps: Thin-plate splines and the decomposition of deformations,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 11, no. 6, pp. 567–585, Jun. 1989.
[9]
F. S. Bragança, C. Roepstorff, M. Rhodin, T. Pfau, P. Van Weeren, and L. Roepstorff, “Quantitative lameness assessment in the horse based on upper body movement symmetry: The effect of different filtering techniques on the quantification of motion symmetry,” Biomed. Signal Process. Control, vol. 57, 2020, Art. no.
[10]
S. Brahnam, C.-F. Chuang, F. Y. Shih, and M. R. Slack, “Machine recognition and representation of neonatal facial displays of acute pain,” Artif. Intell. Med., vol. 36, no. 3, pp. 211–222, 2006.
[11]
A. M. Bronstein, M. M. Bronstein, and R. Kimmel, Numerical Geometry of Non-Rigid Shapes. Berlin, Germany: Springer, 2008.
[12]
S. Broomé, K. Ask, M. Rashid, P. H. Andersen, and H. Kjellström, “Sharing pain: Using domain transfer between pain types for recognition of sparse pain expressions in horses,” 2021,.
[13]
S. Broomé, K. B. Gleerup, P. H. Andersen, and H. Kjellstrom, “Dynamics are important for the recognition of equine pain in video,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 12659–12668.
[14]
C. C. Caeiro, A. M. Burrows, and B. M. Waller, “Development and application of CatFACS: Are human cat adopters influenced by cat facial expressions?,” Appl. Animal Behav. Sci., vol. 189, pp. 66–78, 2017.
[15]
T. J. Cashman and A. W. Fitzgibbon, “What shape are dolphins? Building 3D morphable models from 2D images,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 232–244, Jan. 2013.
[16]
E. Dalla Costa, M. Minero, D. Lebelt, D. Stucke, E. Canali, and M. C. Leach, “Development of the horse grimace scale (HGS) as a pain assessment tool in horses undergoing routine castration,” PLoS One, vol. 9, no. 3, 2014, Art. no.
[17]
J. Egede, M. Valstar, and B. Martinez, “Fusing deep learned and hand-crafted features of appearance, shape, and dynamics for automatic pain estimation,” in Proc. 12th IEEE Int. Conf. Autom. Face Gesture Recognit., 2017, pp. 689–696.
[18]
R. Ekman, What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System. New York, NY, USA: Oxford Univ. Press, 1997.
[19]
K. B. Gleerup, B. Forkman, C. Lindegaard, and P. H. Andersen, “An equine pain face,” Vet. Anaesth. Analg., vol. 42, no. 1, pp. 103–114, 2015.
[20]
K. Gleerup and C. Lindegaard, “Recognition and quantification of pain in horses: A tutorial review,” Equine Vet. Edu., vol. 28, no. 1, pp. 47–57, 2016.
[21]
T. Hadjistavropoulos et al., “Pain assessment in elderly adults with dementia,” Lancet Neurol., vol. 13, no. 12, pp. 1216–1227, 2014.
[22]
T. Hassan et al., “Automatic detection of pain from facial expressions: A survey,” IEEE Trans. Pattern Anal. Mach. Int., vol. 43, no. 6, pp. 1815–1831, Jun. 2021.
[23]
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015,. [Online]. Available: http://arxiv.org/abs/1512.03385
[24]
H. Hummel, “Analysing horse and donkey faces for measuring pain expressions,” MSc thesis, Vrije Universiteit Utrecht, The Netherlands, 2020.
[25]
H. I. Hummel, F. Pessanha, A. A. Salah, T. J. P. A. M. van Loon, and R. C. Veltkamp, “Automatic pain detection on horse and donkey faces,” in Proc. 15th IEEE Int. Conf. Autom. Face Gesture Recognit., 2020, pp. 793–800.
[26]
R. Irani et al., “Spatiotemporal analysis of RGB-DT facial images for multimodal pain level recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2015, pp. 88–95.
[27]
B. Jonkers, “Equine Utrecht University scale for automated recognition in facial assessment of pain – EQUUS-ARFAP,” MSc thesis, Dept. Inf. Comput. Sci., Universiteit Utrecht, Utrecht, The Netherlands, 2018.
[28]
A. Jourabloo and X. Liu, “Large-pose face alignment via CNN-based dense 3D model fitting,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 4188–4196.
[29]
S. Kaltwang, O. Rudovic, and M. Pantic, “Continuous pain intensity estimation from facial expressions,” in Proc. Int. Symp. Vis. Comput., 2012, pp. 368–377.
[30]
A. Kanazawa, S. Kovalsky, R. Basri, and D. Jacobs, “Learning 3D deformation of animals from 2D images,” Comput. Graph. Forum, vol. 35, no. 2, pp. 365–374, 2016.
[31]
V. Kazemi and J. Sullivan, “One millisecond face alignment with an ensemble of regression trees,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 1867–1874.
[32]
S. C. Keating, A. A. Thomas, P. A. Flecknell, and M. C. Leach, “Evaluation of EMLA cream for preventing pain during tattooing of rabbits: Changes in physiological, behavioural and facial expression responses,” PLoS One, vol. 7, no. 9, 2012, Art. no.
[33]
D. J. Langford et al., “Coding of facial expressions of pain in the laboratory mouse,” Nature Methods, vol. 7, no. 6, pp. 447–449, 2010.
[34]
G. C. Lencioni, R. V. de Sousa, E. J. de Souza Sardinha, R. R. Corrêa, and A. J. Zanella, “Pain assessment in horses using automatic facial expression recognition through deep learning-based modeling,” PLoS One, vol. 16, no. 10, 2021, Art. no.
[35]
K. Levenberg, “A method for the solution of certain non-linear problems in least squares,” Quart. Appl. Math., vol. 2, no. 2, pp. 164–168, 1944.
[36]
P. Lucey et al., “Automatically detecting pain in video through facial action units,” IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 3, pp. 664–674, Jun. 2011.
[37]
P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, and I. Matthews, “Painful data: The UNBC-McMaster shoulder pain expression archive database,” in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2011, pp. 57–64.
[38]
M. Mahmoud, Y. Lu, X. Hou, K. McLennan, and P. Robinson, “Estimation of pain in sheep using computer vision,” in Handbook of Pain and Palliative Care. Berlin, Germany: Springer, 2018, pp. 145–157.
[39]
D. W. Marquardt, “An algorithm for least-squares estimation of nonlinear parameters,” J. Soc. Ind. Appl. Math., vol. 11, no. 2, pp. 431–441, 1963.
[40]
B. J. Matuszewski, W. Quan, and L.-K. Shark, “High-resolution comprehensive 3-D dynamic database for facial articulation analysis,” in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2011, pp. 2128–2135.
[41]
K. M. McLennan, C. J. Rebelo, M. J. Corke, M. A. Holmes, M. C. Leach, and F. Constantino-Casas, “Development of a facial expression scale using footrot and mastitis as models of pain in sheep,” Appl. Animal Behav. Sci., vol. 176, pp. 19–26, 2016.
[42]
S. Mehdi and V. Mohammad, “A farm-based prospective study of equine colic incidence and associated risk factors,” J. Equine Vet. Sci., vol. 26, no. 4, pp. 171–174, 2006.
[43]
M. A. Nicolaou, H. Gunes, and M. Pantic, “Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space,” IEEE Trans. Affect. Comput., vol. 2, no. 2, pp. 92–105, Apr.–Jun. 2011.
[44]
A. Noor, Y. Zhao, A. Koubâa, L. Wu, R. Khan, and F. Y. Abdalla, “Automated sheep facial expression classification using deep transfer learning,” Comput. Electron. Agriculture, vol. 175, 2020, Art. no.
[45]
F. Pessanha, M. Mahmoud, and K. McLennan, “Towards automatic monitoring of disease progression in sheep: A hierarchical model for sheep facial expressions analysis from video,” in Proc. 15th IEEE Int. Conf. Autom. Face Gesture Recognit., 2020, pp. 387–393.
[46]
J. Price, J. Marques, E. Welsh, and N. Waran, “Pilot epidemiological study of attitudes towards pain in horses,” Vet. Rec., vol. 151, no. 19, pp. 570–575, 2002.
[47]
S. N. Raja et al., “The revised IASP definition of pain: Concepts, challenges, and compromises,” Pain, vol. 161, no. 9, 2020, Art. no.
[48]
M. Rashid et al., “Equine pain behavior classification via self-supervised disentangled pose representation,” in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis., 2022, pp. 152–162.
[49]
M. Rashid, A. Silventoinen, K. B. Gleerup, and P. H. Andersen, “Equine facial action coding system for determination of pain-related facial responses in videos of horses,” PLoS One, vol. 15, no. 11, 2020, Art. no.
[50]
N. Ruiz, E. Chong, and J. M. Rehg, “Fine-grained head pose estimation without keypoints,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2018, pp. 2155–215509.
[51]
A. A. Salah, N. Sebe, and T. Gevers, “Communication and automatic interpretation of affect from facial expressions,” in Affective Computing and Interaction: Psychological, Cognitive and Neuroscientific Perspectives. Hershey, PA, USA: IGI Global, 2011, pp. 157–183.
[52]
S. G. Sotocina et al., “The rat grimace scale: A partially automated method for quantifying pain in the laboratory rat via facial expressions,” Mol. Pain, vol. 7, pp. 1744–8069, 2011.
[53]
A. H. Tuttle et al., “A deep neural network to assess spontaneous pain from mouse facial expressions,” Mol. Pain, vol. 14, 2018, Art. no.
[54]
J. P. van Loon and L. Macri, “Objective assessment of chronic pain in horses using the horse chronic pain scale (HCPS): A scale-construction study,” Animals, vol. 11, no. 6, 2021, Art. no.
[55]
J. P. van Loon and M. C. Van Dierendonck, “Monitoring acute equine visceral pain with the Equine Utrecht University scale for composite pain assessment (EQUUS-COMPASS) and the equine Utrecht University scale for facial assessment of pain (EQUUS-FAP): A scale-construction study,” Vet. J., vol. 206, no. 3, pp. 356–364, 2015.
[56]
J. P. van Loon and M. C. Van Dierendonck, “Monitoring equine head-related pain with the Equine Utrecht University scale for facial assessment of pain (EQUUS-FAP),” Vet. J., vol. 220, pp. 88–90, 2017.
[57]
S. Walter et al., “The BioVid heat pain database data for the advancement and systematic validation of an automated pain recognition system,” in Proc. IEEE Int. Conf. Cybern., 2013, pp. 128–131.
[58]
J. Wathan, A. M. Burrows, B. M. Waller, and K. McComb, “EquiFACS: The equine facial action coding system,” PLoS One, vol. 10, no. 8, pp. 1–35, 2015.
[59]
P. Werner, D. Lopez-Martinez, S. Walter, A. Al-Hamadi, S. Gruss, and R. Picard, “Automatic recognition methods supporting pain assessment: A survey,” IEEE Trans. Affect. Comput., vol. 13, no. 1, pp. 530–552, Jan.–Mar. 2022.
[60]
X. Xiong and F. De la Torre, “Supervised descent method and its applications to face alignment,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 532–539.
[61]
X. Zhang et al., “BP4D-spontaneous: A high-resolution spontaneous 3D dynamic facial expression database,” Image Vis. Comput., vol. 32, no. 10, pp. 692–706, 2014.
[62]
X. Zhu, Z. Lei, X. Liu, H. Shi, and S. Z. Li, “Face alignment across large poses: A 3D solution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 146–155.
[63]
S. Zuffi, A. Kanazawa, D. W. Jacobs, and M. J. Black, “3D menagerie: Modeling the 3D shape and pose of animals,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 5524–5532.
[64]
W. Zwitserloot, “Quantification of pain from facial expression characteristics assessed from photos of horses and donkeys,” MSc thesis, Dept. Equine Sci., Universiteit Utrecht, Utrecht, The Netherlands, 2019.
Recommendations
Automatic Pain Assessment with Facial Activity Descriptors
Pain is a primary symptom in medicine, and accurate assessment is needed for proper treatment. However, today’s pain assessment methods are not sufficiently valid and reliable in many cases. Automatic recognition systems may contribute to overcome ...
Machine assessment of neonatal facial expressions of acute pain
We propose that a machine assessment system of neonatal expressions of pain be developed to assist clinicians in diagnosing pain. The facial expressions of 26 neonates (age 18-72h) were photographed experiencing the acute pain of a heel lance and three ...
Assessment of Pain Using Facial Pictures Taken with a Smartphone
COMPSAC '15: Proceedings of the 2015 IEEE 39th Annual Computer Software and Applications Conference - Volume 02Timely and accurate information about patients' symptoms is important for clinical decision making such as adjustment of medication. Due to the limitations of self-reported symptom such as pain, we investigated whether facial images can be used for ...
Comments
Information & Contributors
Information
Published In
1949-3045 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Computer Society Press
Washington, DC, United States
Publication History
Published: 01 July 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Nov 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in