Automatic Labels are as Effective as Manual Labels in Biomedical Images Classification with Deep Learning

The development of deep learning (DL) algorithms is fostering the design of new tools that can be trained on clinical data without the need for human intervention, especially in domains where the cost of annotations is high, such as histopathology. Histopathology is the gold standard to diagnose cancer (Van der Laak et al., 2021; De Matos et al., 2021). The domain involves the analysis of small tissue slices, to identify microscopic findings related to dangerous diseases (Gurcan et al., 2009), such as cancer. Tissue slices undergo microscopic examination by a medical expert named pathologist, who usually needs up to an hour per image to analyze a single sample (Krupinski et al., 2013). Despite the increasing digitization of tissue samples, histopathological samples are still rarely analyzed exploiting digital aid in clinical practice (Fraggetta et al., 2017, 2021). Digital pathology is a domain involving the management and digitization of tissue specimens, called Whole Slide Images (WSI). WSIs are high-resolution images stored with a pyramidal format, to capture different magnification levels of details (Merchant and Castleman, 2022). Usually, the highest resolution levels result in a spatial high-resolution of 0.25–0.5$\mu$m per pixel, which corresponds to an optical resolution of 20-40x. WSIs are usually coupled with pathology reports. Pathology reports are semi-structured free-text documents containing information about the patient anamnesis, the tissue specimen type and the findings and observations identified by a pathologist during the tissue examination (Hewer, 2020; Hanna et al., 2020). WSIs and reports are usually stored in the Laboratory Information System (LIS), easily enabling sample retrieval. The increasing collection of biomedical samples is encouraging the design of automatic tools to analyze WSIs under the computational pathology domain (Van der Laak et al., 2021; Madabhushi and Lee, 2016; Litjens et al., 2022). Most of the computational domain algorithms are currently based on deep learning, such as CNNs (Convolutional Neural Networks) or ViT (Visual Transformers) (Xu et al., 2023; Cifci et al., 2023).

Even if computational pathology algorithms show accurate and robust performance, in tasks such as WSI classification or segmentation, several challenges are still open, such as data labels (Madabhushi and Lee, 2016; Campanella et al., 2019; Van der Laak et al., 2021; Abels et al., 2019; Chen et al., 2022). Data labels are required to train supervised learning algorithms. However, the collection of labels is not trivial, considering both strong and weak annotations. Even if strong labels (i.e. pixel-wise annotations) usually achieve the most accurate performance when used to train a deep learning model, they require a pathologist to analyze samples, which can be time-consuming, so often unfeasible (Karimi et al., 2020). Therefore, the research based on the analysis of WSIs is mostly based on the exploitation of weak (i.e. image-level) labels. Weak labels are related to the global image, even if they originate from a region of the image including specific characteristics, such as cancer (Deng et al., 2020). Weak labels are inherently more noisy than pixel-wise annotations, since the regions leading to a specific label may be a small percentage of the whole image (e.g. 1-2%). For this reason, algorithms based on weak labels require larger training datasets to reach accurate performance. Currently, most of weakly-supervised algorithms in computational pathology are based on Multiple Instance Learning (MIL) framework (Carbonneau et al., 2018), which models the whole image as a bag of instances, where only the annotations about the global image are available. MIL framework includes several algorithms, which lately showed high performance when adopted on large-scale datasets (Campanella et al., 2019; Ilse et al., 2018; Wang et al., 2019; Lu et al., 2021; Hashimoto et al., 2020; Chen et al., 2022). For example, Campanella et al. (2019) showed that it is possible to reach almost perfect predictions on binary classification (cancer vs. non-cancer) using around 10’000 weakly-annotated WSIs, on three use cases: skin, breast, and prostate images. The production of weak labels is faster than the strong ones, since they can be extracted from reports. For example, the analysis of a report may require approximately 30 seconds/1 minute, in comparison with the analysis of an image, which requires around an hour. However, human intervention is usually still required to analyze reports, unless the Laboratory Information System (LIS), where the samples and corresponding reports are stored, has a specific structure to retrieve automatically data, according to the characteristics that can be used as labels. Unfortunately, most LISs do not show this feature, since they are organized in heterogeneous ways.

Automatic methods for extracting concepts from reports and using them as weak labels already exist (Marini et al., 2022), but noisy characteristics of weak labels can make automatic labeling ineffective. This paper aims to investigate under which circumstances automatic labels (i.e. labels automatically generated by an algorithm) can be adopted to train deep learning models to alleviate the need for experts to annotate data. In particular, the goal is to identify when the results achieved using this type of labels reach results comparable to the ones obtained using manual labels (i.e. labels produced by a medical expert), so that data included in LISs can be fully exploited to build more robust and accurate tools to diagnose diseases. The characteristics investigated in the paper involve the percentage of wrongly automatic labels necessary to reach comparable performance obtained with manual labels, the nature of labels (i.e. binary, multiclass and multilabel) and the deep learning architecture (robust or less robust to noise). Wrongly automatic labels are annotations that are automatically produced by an algorithm and do not match the ground truth (i.e. manually-made).

The paper includes a comparison of deep learning architecture trained with automatic and manual labels on the classification of WSIs. The comparison involves two sets of experiments: a controlled scenario and a real-case scenario. In the controlled scenario, manual labels are randomly perturbed with different percentages of noise, simulating the output of an algorithm to generate automatic labels. The random perturbation involves a modification on the labels: in the celiac disease use case (binary), labels are flipped; in the lung cancer use case (multiclass), a different class is assigned to a sample; in the colon cancer use case (multilabel), labels are modified so that one or more classes from the original label are flipped. In the real-case scenario, the Semantic Knowledge Extractor Tool (SKET) (Marchesin et al., 2022) is used to extract meaningful concepts from reports that are used as weak labels for the corresponding samples.

The analysis involves three tissue use cases, celiac disease, lung cancer and colon cancer, composing a training dataset with over 10’000 WSIs, used to train three deep learning architectures: CLAM (Lu et al., 2021), transMIL (Shao et al., 2021) and Vision Transformer (ViT) (Chen et al., 2022).

Celiac disease (CD) is an autoimmune disorder leading to inflammations and damage in the small intestine, resulting in a range of gastrointestinal and systemic symptoms (Caio et al., 2019). Globally, celiac disease affects about 1-2% of the population (Lebwohl and Rubio-Tapia, 2021), with variations across regions. Celiac disease diagnosis involves duodenal biopsies and serological tests (for specific antibodies). In particular, the examination of biopsies aims to identify villous atrophy, crypt hyperplasia and increased intraepithelial lymphocytes. In this paper, duodenal samples are labeled with celiac disease or normal tissue (binary labels).

Lung cancer is the leading cause of death related to cancer worldwide (Schabath and Cote, 2019; Organization, 2023). It is categorized into two main primary groups: Non-Small Cell Lung Cancer (NSCLC), which represents the large majority of cases (about 85% of cases), and Small-Cell Lung Cancer (SCLC), which is less common, but more aggressive. Furthermore, NSCLC is further described with subtypes, such as LUng ADenocarcinoma (LUAD), LUng Squamous cell Carcinoma (LUSC). Diagnosis of lung cancer through biopsies often involves the identification of irregular cell patterns, architectural distortion, and increased cellular density (Travis, 2011). In this paper, lung samples are labeled with SCLC, LUAD, LUSC, Normal Tissue.

Colon cancer is the fourth most often diagnosed cancer worldwide (Benson et al., 2018). Colon cancer diagnosis involves the identification of multiple concepts, such as the presence of cancer and the evaluation of polyp shapes and possible abnormalities leading to dysplasia. In this paper, colon samples are labeled with colon cancer, high-grade dysplasia (HGD), low-grade dysplasia (LGD), hyperplastic polyp and normal tissue (multilabel labels). Figure 1 shows some histopathological samples corresponding to the three tissues.

The dataset used in this paper includes WSIs and reports (paired together) of celiac disease, lung cancer and colon cancer, collected from two hospitals: the Catania cohort and Radboudumc (RUMC).

WSIs are used to train and test different computer vision architectures on the image-level classification. WSIs are gigapixel tissue samples that can exhibit significant heterogeneity, for example in terms of staining (Marini et al., 2021a, 2023) and sample types. The image heterogeneity is a consequence of different acquisition procedures across laboratories, related to the chemical reagents applied to the specimen and to the whole slide scanners. One of the main consequences of the heterogeneity is the stain variability, leading to different color variations, intensity, and uniformity of stains across different slides (as shown in Figure 1). Also, the WSIs collected in this dataset show the same characteristic, aiming to replicate a common scenario in digital pathology. WSIs collected from the Catania cohort were scanned with two 3DHistech scanners and two Aperio scanners and stored with a magnification of 20-40x; WSIs collected from RUMC were scanned using 3DHistech scanners, mainly stored at 40x magnification.

Reports are used to extract meaningful concepts, that are used as weak automatic labels to train the model to classify WSIs. Reports include free-text descriptions summarizing the findings from tissue examination. The findings are reported in a field named ‘Conclusion’, containing either macroscopic or microscopic observations. Even if a report includes many fields, only the findings are relevant for the analysis proposed in the paper. Therefore, additional patient information, such as family history or personal data, is discarded. Textual reports show heterogeneity, mainly related to the source language and the textual content. Reports are collected from an Italian and a Dutch hospital, therefore they have to be translated into English, to standardize the analysis. The textual content slightly differs across sources, because the Catania cohort reports contain a field specifically for the findings identified in a single slide, while the RUMC reports include a specific field for the findings identified in a tissue block, which may encompass multiple slides. Furthermore, samples are collected across years and are produced by many different pathologists, each one adopting its unique style of writing.

The dataset includes samples collected from three different use cases: celiac disease, lung cancer, colon cancer. Data are randomly selected from LISs, to simulate a real-case scenario. The goal is to show that the approach can generalize on different types of tissue (both in terms of images and reports). Furthermore, different types of labels are used: celiac disease samples are annotated with binary labels, lung samples with multiclass labels, and colon samples with multilabel samples.

Table 1 includes a detailed composition of data related to celiac disease collected from pathology reports, split into training and testing partitions. Data are labeled with binary labels: celiac disease and normal tissue.

Table 2 includes a detailed composition of data related to lung cancer collected from pathology reports, split in training and testing partitions. Data are labeled with multiclass labels: Small-Cell Cancer, Non-Small Adenocarcinoma Cell Cancer, Non-Small Squamous Cell Cancer, Normal Tissue.

Table 3 includes a detailed composition of data related to colon cancer collected from pathology reports, split into training and testing partitions. Data are labeled with multilabel labels: Adenocarcinoma, High-Grade Dysplasia (HGD), Low-Grade Dysplasia (LGD), Hyperplastic Polyp, Normal Tissue.

The training schema is based on computer vision algorithms to classify WSIs, comparing the performance of automatic and manual labels during the training. Those algorithms are based on weak labels, since they are easier to be collected, even if they still require the intervention of medical experts. In this paper, three different MIL backbones are adopted: two CNNs, CLAM and transMIL, and a ViT. The architectures are trained to evaluate the effect that automatic labels may have on the training of models to classify WSIs. Firstly, they are trained with noisy labels, randomly generated perturbating the manual labels, with a different percentage (1,2,5,10,20,50%) of noise. The goal of this experiment is to evaluate the effect that noisy labels have on the performance of a model. However, this setup does not fit a real-case scenario where automatic labels are adopted. Noisy labels may be considered as wrongly-labeled samples, but not all mistakes on labels have the same likelihood to happen. Considering for example weak labels inferred by reports: some reports, due to their content, may be more easily mislabeled. For this reason, a second setup is proposed, adopting a real tool to extract concepts from reports: the Semantic Knowledge Extractor Tool (SKET) (Marchesin et al., 2022). This second experiment also helps to test the rules identified within the first experiment.

Figure 2 shows an overview of the data analysis pipeline.

Three computer vision algorithms to classify WSIs are compared in the paper as backbones, to evaluate the effect that noisy labels may have on different architectures, including two CNNs and a ViT. The CNNs have a ResNet34 backbone, while the ViT has a backbone similar to the one shown in Chen et al. (2022), considering a single magnification level. In both cases, the backbones are designed to output an embedding of size 128 representing a single WSI, so that the same classifier can be adopted for all architectures, modifying the output classes based on the use case.

SKET (Marchesin et al., 2022) is an unsupervised algorithm combining a rule-based expert system with machine learning models, chosen to extract meaningful concepts from reports and use them as weak labels for WSIs (Marchesin et al., 2022; Menotti et al., 2023). The algorithm includes three components: Named Entity Recognition, Entity Linking and Data Labeling. Named Entity Recognition involves pre-trained models (ScispaCy models (Neumann et al., 2019)), developed to work on biomedical data, and large Word2Vec word vectors (Mikolov et al., 2013) trained on the PubMed Central Open Access Subset (Mikolov et al., 2013). Entity Linking involves a combination of similarity matching techniques to match ad-hoc concepts to a reference ontology. Data Labeling involves the mapping of the concepts with a set of annotation classes. SKET is an unsupervised model, therefore no training data are required to tune it. This feature is relevant since it does not require data annotation for training, such as other Natural Language Processing (NLP) algorithms.

Image pre-processing includes the WSI splitting into patches. WSIs usually do not fit modern GPU hardware memory because of their gigapixel characteristics, therefore they have to be split into patches. In this paper, WSIs are split 224x224 pixel patches using the Multi_Scale_Tools library (Marini et al., 2021b). The choice of the size is related to the characteristics of ResNet34 backbone, requiring fixed input size. Patches are extracted from magnification 5x considering celiac samples, while lung and colon patches are sampled from magnification 10x. The magnifications are chosen considering that the magnification allows to identify peculiar morphological features, useful for the classification task: 5x for celiac disease cases, 10x for lung and colon ones. The choice of the magnification to examine is driven by the characteristics of the problem to solve: celiac disease diagnosis requires to identify the villous shape and the crypts, therefore 5x magnification is chosen; on the other hand, lung and colon require a more refined level of magnification, because the shape of glands is as relevant as the cell infiltration, therefore 10x is chosen. Not all sampled patches are selected: the ones from background regions are discarded, being not informative. The identification of background regions involves the application of HistoQC tool (Janowczyk et al., 2019), which generates tissue masks.

The report pre-processing only involves their translation into English. Original reports are stored in Italian and Dutch, depending on the workflow from which they are collected. The translation is necessary because state-of-the-art NLP algorithms are mostly developed to work with inputs in English. MarianMT neural machine translation models (Junczys-Dowmunt et al., 2018) are used to translate the content of the reports to English.

The backbones of deep learning algorithms to analyze images are pre-trained using self-supervised algorithms: simCLR (Chen et al., 2020) for the CNNs (CLAM and transMIL), DINO v2 (Oquab et al., 2023) for the ViT.

Both algorithms are adopted to learn meaningful features from unannotated input data, exploiting similarities and dissimilarities between input samples. In this paper, the input data for the algorithms are the patches sampled from the training partition. Since data are unannotated, no information is available regarding similarity among patches. Therefore, data augmentation is adopted: samples are similar to their augmented versions and dissimilar from the other samples within a batch. The algorithms differ on the data augmentation strategy. simCLR is designed for CNNs and its augmentation pipeline includes several operations, applied with a probability of 0.5: random rotations (90/180/270 degrees), vertical/horizontal flipping, hue-saturation-contrast (HUE) color augmentation, RGB shift, color jitter, gaussian noise, elastic transformation, grid distortions. DINO is designed for ViT and involves a knowledge distillation mechanism: two networks are involved in the training, a teacher and a student. The teacher is a larger model producing outputs that the student aims to mimic and replicate. Both models are directly trained with two different augmented versions of input samples. However, the student is also trained with a cropped version (96x96 pixels) of the teacher inputs. DINO v2 augmentation pipeline includes two augmentation pipelines: the first one includes color jitter, horizontal/vertical flipping, gaussian blur; the second one includes color jitter, horizontal/vertical flipping, gaussian blur, solarization.

Albumentations library (Buslaev et al., 2020) is adopted to apply data augmentation to input images. The operations involved are random rotations (90/180/270 degrees), vertical/horizontal flipping and hue-saturation-contrast (HUE) color augmentation. The operations from the data augmentation pipeline are selected with a probability of 0.5 and applied at image-level, so that all the patches are augmented consistently

The performance of the models is evaluated in terms of WSI classification, using the weighted F1-score. The classification problem can be defined as a binary problem (celiac disease), multiclass problem (lung cancer) or multilabel problem (colon cancer). F1-score is a metric to measure the accuracy of a classifier, combining the recall and the precision. Precision evaluates how well a classifier is robust to avoid predicting negative samples as positive ones, while recall evaluates how well it correctly classifies all the positive samples. In all the use cases, data may show unbalanced class distribution, since they are randomly selected from workflows, aiming to simulate a real-case scenario. For this reason, weighted macro F1-score is adopted Weighted F1-score tackles class imbalance, evaluating the F1-scores for the single classes and then averaging them according the class support (number of true samples for the class). The weighted F1-score is reported in terms of average and standard deviation of the ten experiment repetitions, evaluated on the test partition.

The performance difference among different setups is evaluated through the Wilcoxon Rank-Sum test (Woolson, 2007). The test aims to establish if the results of two different experiments are statistically significantly different ($p$-value < 0.05).

All the setups presented in the paper are trained using $k$-fold cross-validation, in order to evaluate the robustness of the model on data used for training. The training partition is divided into $k$ folders ($k$=10 in this paper). During every training repetition, k-1 folders are used to train the model, while the other group is used to validate it. Data are split in partitions considering the patients, so that WSIs collected from a patient cannot be in two different partitions

The experiments are developed exploiting Python libraries. The deep learning algorithms are implemented and trained using PyTorch 2.2.0 and run on a Tesla V100 GPU. WSIs are accessed using openslide 3.4.1 (Goode et al., 2013). WSI pre-processing involves Multi_Scale_Tools library (Marini et al., 2021b) and data augmentation is applied using albumentations 1.3.1 (Buslaev et al., 2020). The performance of the model is quantitavely evaluated using the metrics implemented by scikit-learn 0.22.

The optimal configuration setup of both CNN and ViT hyperparameters is identified using the grid search algorithm. The optimal set is the one reaching the lowest loss function of the classification of WSIs, considering the validation partition. The parameters tested with the grid search algorithm are: the batch size (4 selected; 1,2,4,8 tested); the CNN optimizer (Adam selected); the ViT optimizer (Adam selected; Adam, LARS and AdamW tested); the number of epochs when the CNN model is trained (15; over this number of epochs, the loss function evaluated on the validation partition no longer decreases); the number of epochs when the HIPT model is trained (15; over this number of epochs, the loss function evaluated on the validation partition no longer decreases); the learning rate ($10^{-4}$; $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ were tested); the decay rate ($10^{-4}$; $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ were tested); the number of nodes in the intermediate layer after the ResNet and the ViT backbone (128; 64, 128, 256, 512 were tested).

Meaningful concepts can be extracted from pathology reports without the need for human intervention and can be adopted as weak labels, dramatically reducing the time needed to collect labels.

The performance of SKET (tool to extract weak labels from reports) is evaluated on the training partition of the three use cases, since SKET does not require any training, being a ruled-based algorithm. The extracted concepts are compared with the manual labels, provided by medical experts. Table 4 summarizes the results. On every use case, SKET reaches a weighted F1-score over 0.944, considering the cumulative testing partition. On the single pathology workflows, the lowest performance is reached considering the Catania testing partition of celiac disease data (0.860). Otherwise, the algorithm reaches high-level performance, always over 0.960 in terms of F1-score.

Being effective, SKET can be adopted to mine unlabeled datasets and to annotate large amounts of data, that can be used to train deep learning models. When tested on a Tesla V100 GPU, SKET requires among 0.006 and 0.03 seconds to extract concepts from a report, depending on its length. Considering the worst case scenario, the algorithm is still a thousand times faster than a human expert who needs in the best case scenario 30 s per report. Therefore, the application of SKET leads to save 99.99% of time required in comparison with human experts. For instance, the weak labeling of 10’000 WSIs would require 300’000 seconds (around 83 hours, without breaks) for human experts, in the best case scenario; on the other hand, it would require 300 seconds (five minutes), in the worst case scenario, to SKET.

The classification performance of multiple computer vision architectures trained with binary automatically annotated data to classify celiac disease WSIs is as effective as the performance reached by models using manually annotated data.

Tables 5 and 6 summarize the results. The highest performance using manual labels is reached using a ViT architecture (F1-score = 0.914 $\pm$ 0.014 on the test partition), even if on the Catania partition transMIL shows the highest performance. The results are still similar for the three architectures.

Table 5 shows the classification performance obtained using binary manual labels and noisy labels. This experiment aims to investigate general rules for the adoption of automatic labels on the binary classification of WSIs. Considering all the architectures, the performance is similar to the one obtained using manual labels, especially until 10% of training samples are wrongly-annotated, the difference in terms of performance is not statistically significant. When the percentage of wrongly-annotated training is 20% (or more) the performance degrades and the difference, compared with manual labels, is statistically significant, suggesting this percentage of wrongly-annotated labels can be considered as a threshold for the adopting of automatic weak labels in a binary classification scenario.

Table 6 shows the comparison of automatic labels, generated with SKET, and manual labels. The comparison among automatic and manual labels shows a F1-score equal to 0.944, suggesting that the algorithm should lead to performance similar to the one obtained with noisy labels when the percentage of mislabeled data is between 2% and 5%. The results confirm the hypothesis, since the performance is slightly worse than the one obtained using manual labels, but the gap is not statistically significant (according to the Wilcoxon Rank-Sum test, comparing every setup to the one where manual labels are used), showing the effectiveness of automatic labels in a binary classification scenario.

The classification performance of multiple computer vision architectures trained with multiclass automatically annotated data to classify lung cancer WSIs is as effective as the performance reached by models using manually annotated data.

Tables 7 and 8 summarize the results. The highest performance using manual labels is reached using a ViT architecture (F1-score = 0.763 $\pm$ 0.012) on both test partitions, dramatically outperforming the other two architectures (CLAM reaches 0.674 $\pm$ 0.016, while transMIL reaches 0.696 $\pm$ 0.016).

Table 7 shows the classification performance obtained using multiclass manual labels and noisy labels. This experiment aims to investigate general rules for the adoption of automatic labels on the multiclass classification of WSIs. Considering all the architectures, the performance is similar to the one obtained using manual labels, especially until 20% of training samples are wrongly-annotated, the difference in terms of performance is not statistically significant. When the percentage of wrongly-annotated training is 50% the performance degrades and the difference, compared with manual labels, is statistically significant, suggesting this percentage of wrongly annotated labels can be considered as a threshold for the adoption of automatic weak labels in a multiclass classification scenario.

Table 8 includes the comparison of automatic labels and manual labels. This comparison represents a real-case scenario of automatic data labeling, where automatic labels are generated by extracting concepts from reports. The comparison among labels shows a F1-score equal to 0.976, suggesting that the algorithm should lead to performance similar to the one obtained in the previous experiment using 2% and 5%. The results confirm the hypothesis, since the performance is slightly worse than the one obtained using manual labels, but the gap is not statistically significant (according to the Wilcoxon Rank-Sum test, comparing every setup to the one where manual labels are used).

The classification performance of multiple computer vision architectures trained with multilabel automatically annotated data to classify colon cancer WSIs is as effective as the performance reached by models using manually annotated data.

Tables 9 and 10 summarize the results. The highest performance using manual labels is reached using a ViT architecture (F1-score = 0.831 $\pm$ 0.009) on both test partitions, dramatically outperforming the other two architectures (CLAM reaches 0.773 $\pm$ 0.015, while transMIL reaches 0.791 $\pm$ 0.008).

Table 9 shows the classification performance obtained using multilabel manual labels and noisy labels. This experiment aims to investigate general rules for the adoption of automatic labels on the multilabel classification of WSIs. Considering all the architectures, the performance is similar to the one obtained using manual labels, especially until 20% of training samples are wrongly-annotated, the difference in terms of performance is not statistically significant. When the percentage of wrongly-annotated training is 50% the performance degrades and the difference, compared with manual labels, is statistically significant, suggesting this percentage of wrongly-annotated labels can be considered as a threshold for the adoption of automatic weak labels in a multilabel classification scenario.

Table 10 includes the comparison of automatic labels and manual labels. This comparison represents a real-case scenario of automatic data labeling, where automatic labels are generated extracting concepts from reports. The comparison among labels shows a F1-score equal to 0.971, suggesting that the algorithm should lead to performance similar to the one obtained in the previous experiment using 2% and 5%. The results confirm the hyphotesis, since the performance are slightly worse than the one obtained using manual labels, but the gap is not statistically significant (according to Wilcoxon Rank-Sum test, comparing every setup to the one where manual labels are used).