Vision Transformer Model for Automated End-to-End Radiographic Assessment of Joint Damage in Psoriatic Arthritis
Authors: 
Darshana Govind, Zijun Gao, Chaitanya Parmar, Kenneth Broos, Nicholas Fountoulakis, Lenore Noonan, Shinobu Yamamoto, Natalia Zemlianskaia, Craig S. Meyer, Emily Scherer, Michael Deman, Pablo Damasceno, Philip S. Murphy, Terence Rooney, Elizabeth Hsia, Anna Beutler, Robert Janiczek, Stephen S. F. Yip, Kristopher StandishAuthors Info & Claims
Darshana Govind, Zijun Gao, Chaitanya Parmar, Kenneth Broos, Nicholas Fountoulakis, Lenore Noonan, Shinobu Yamamoto, Natalia Zemlianskaia, Craig S. Meyer, Emily Scherer, Michael Deman, Pablo Damasceno, Philip S. Murphy, Terence Rooney, Elizabeth Hsia, Anna Beutler, Robert Janiczek, Stephen S. F. Yip, Kristopher StandishAuthors Info & ClaimsPages 94 - 103
Abstract
Deep learning techniques such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have demonstrated strong capabilities in region-of-interest (ROI) detection and disease severity scoring, aiming to reduce the manual workload for clinicians. However, studies encompassing an end-to-end method for ROI detection, their subsequent scoring, and aggregation to a patient level score in psoriatic arthritis (PsA) have been limited. To address this gap, we have developed an end-to-end multistep approach for automated radiographic assessment in PsA that detects joints-of-interest from X-rays, assigns structural damage scores to the identified joints, and aggregates them to an extremity level, and subsequently to a patient level score. Furthermore, we compare our approach with state-of-the-art methods and human experts. Our results demonstrate the strong performance of our joint detection models, achieving an average intersection over union (IoU) value of 0.88 for foot joints and 0.76 for hand joints. The subsequent scoring models for structural damage show excellent (intra-class correlation coefficient (ICC) > 0.90) agreement with an ICC of 0.94, 0.98, and 0.97 for joint level erosion, JSN, and patient level scores, respectively, when compared to human readers.
References
[1]
Rana M and Bhushan M Machine learning and deep learning approach for medical image analysis: diagnosis to detection Multimedia Tools Appl. 2023 82 17 26731-26769
[2]
Jiang, H., et al.: A review of deep learning-based multiple-lesion recognition from medical images: classification, detection and segmentation. Comput. Biol. Med. 106726 (2023)
[3]
Imtiaz, M., Shah, S.A.A., ur Rehman, Z.: A review of arthritis diagnosis techniques in artificial intelligence era: current trends and research challenges. Neurosci. Inform. 2(4), 100079 (2022)
[4]
Parashar A et al. Medical imaging in rheumatoid arthritis: a review on deep learning approach Open Life Sci. 2023 18 1 20220611
[5]
McMaster C et al. Artificial intelligence and deep learning for rheumatologists Arthrit. Rheumatol. 2022 74 12 1893-1905
[6]
Van der Heijde D How to read radiographs according to the Sharp/van der Heijde method J. Rheumatol. 2000 27 1 261-263
[7]
Rahman P et al. Radiological assessment in psoriatic arthritis Br. J. Rheumatol. 1998 37 7 760-765
[8]
Montalbo, F.J.P.: A computer-aided diagnosis of brain tumors using a fine-tuned YOLO-based model with transfer learning. KSII Trans. Internet Inf. Syst. 14(12) (2020)
[9]
Liu K Stbi-yolo: a real-time object detection method for lung nodule recognition IEEE Access 2022 10 75385-75394
[10]
Glenn Jocher, A.C., Qiu, J.: Ultralytics YOLO. 2023 [cited 2024 1]; 8.0.0:[29]. Available from: https://github.com/ultralytics/ultralytics
[11]
Zhao X et al. A cross-modal 3D deep learning for accurate lymph node metastasis prediction in clinical stage T1 lung adenocarcinoma Lung Cancer 2020 145 10-17
[12]
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
[13]
Gheflati, B., Rivaz, H.: Vision transformers for classification of breast ultrasound images. In: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE (2022)
[14]
Matsoukas, C., et al.: Is it time to replace CNNs with transformers for medical images? arXiv preprint arXiv:2108.09038, 2021
[15]
Huang, Y.-J., et al.: Automatic joint space assessment in hand radiographs with deep learning among patients with rheumatoid arthritis. In: Arthritis & Rheumatology. 2020. Wiley 111 RIVER ST, HOBOKEN 07030–5774, NJ USA
[16]
Radke KL et al. Adaptive IoU thresholding for improving small object detection: a proof-of-concept study of hand erosions classification of patients with rheumatic arthritis on X-ray images Diagnostics 2022 13 1 104
[17]
Stolpovsky, A., et al.: RheumaVIT: transformer-based model for Automated Scoring of Hand Joints in Rheumatoid Arthritis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
[18]
Wang H-J et al. Deep learning-based Computer-Aided diagnosis of rheumatoid arthritis with hand X-ray images Conforming to Modified total Sharp/van der Heijde score Biomedicines 2022 10 6 1355
[19]
Li H and Guan Y Multilevel modeling of joint damage in rheumatoid arthritis Adv. Intell. Syst. 2022 4 11 2200184
[20]
Rohrbach J et al. Bone erosion scoring for rheumatoid arthritis with deep convolutional neural networks Comput. Electr. Eng. 2019 78 472-481
[21]
Saalfeld, W., et al.: Differentiating psoriatic arthritis from osteoarthritis and rheumatoid arthritis: a narrative review and guide for advanced practice providers. Rheumatol. Therapy 8, 1–25 (2021)
[22]
Koo TK and Li MY A guideline of selecting and reporting intraclass correlation coefficients for reliability research J. Chiropr. Med. 2016 15 2 155-163
[23]
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part V. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014).
[24]
Oquab, M., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
[25]
Khan S et al. Transformers in vision: a survey ACM Comput. Surv. (CSUR) 2022 54 10s 1-41
[26]
He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)
[27]
Huang, G., et al.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
Index Terms
- Vision Transformer Model for Automated End-to-End Radiographic Assessment of Joint Damage in Psoriatic Arthritis
Index terms have been assigned to the content through auto-classification.
Recommendations
A framework for longitudinal latent factor modelling of treatment response in clinical trials with applications to Psoriatic Arthritis and Rheumatoid Arthritis
Abstract Objective:Clinical trials involve the collection of a wealth of data, comprising multiple diverse measurements performed at baseline and follow-up visits over the course of a trial. The most common primary analysis is restricted to a single, ...
Graphical abstractDisplay Omitted
Self-supervised deep learning for joint 3D low-dose PET/CT image denoising
AbstractDeep learning (DL)-based denoising of low-dose positron emission tomography (LDPET) and low-dose computed tomography (LDCT) has been widely explored. However, previous methods have focused only on single modality denoising, neglecting the ...
Graphical abstractDisplay Omitted
Highlights- A two-stage self-supervised framework named MAsk-then-Cycle (MAC) is proposed to enable self-supervised denoising of LDPET/LDCT images. MAC includes a masked autoencoder-based pre-training stage and a self-supervised denoising training ...
End-to-End Deep Learning for Driver Distraction Recognition
Image Analysis and RecognitionAbstractIn this paper, an end-to-end deep learning solution for driver distraction recognition is presented. In the proposed framework, the features from pre-trained convolutional neural networks VGG-19 are extracted. Despite the variation in illumination ...
Comments
Information & Contributors
Information
Published In
Oct 2024
432 pages
ISBN:978-3-031-73283-6
DOI:10.1007/978-3-031-73284-3
- Editors:
- Xuanang Xu,
- Zhiming Cui,
- Islem Rekik,
- Xi Ouyang,
- Kaicong Sun
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 23 October 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 30 Jan 2025