Enhancing the Utility of Privacy-Preserving Cancer Classification using Synthetic Data

Abstract

Deep learning holds immense promise for aiding radiologists in breast cancer detection. However, achieving optimal model performance is hampered by limitations in availability and sharing of data commonly associated to patient privacy concerns. Such concerns are further exacerbated, as traditional deep learning models can inadvertently leak sensitive training information. This work addresses these challenges exploring and quantifying the utility of privacy-preserving deep learning techniques, concretely, (i) differentially private stochastic gradient descent (DP-SGD) and (ii) fully synthetic training data generated by our proposed malignancy-conditioned generative adversarial network. We assess these methods via downstream malignancy classification of mammography masses using a transformer model. Our experimental results depict that synthetic data augmentation can improve privacy-utility tradeoffs in differentially private model training. Further, model pretraining on synthetic data achieves remarkable performance, which can be further increased with DP-SGD fine-tuning across all privacy guarantees. With this first in-depth exploration of privacy-preserving deep learning in breast imaging, we address current and emerging clinical privacy requirements and pave the way towards the adoption of private high-utility deep diagnostic models. Our reproducible codebase is publicly available at https://github.com/RichardObi/mammo_dp.

Figure 1: Overview of our privacy-preserving deep learning pipeline and malignancy-conditioned generative adversarial network (MCGAN).

1 Introduction

Breast cancer accounts for staggering estimates of 684.000 deaths and 2,26 million new cases worldwide per year [11]. Part of this burden could be reduced through earlier detection and timely treatment. Screening mammography is a cornerstone for early detection and further associated with a reduction in breast cancer mortality [20]. Recent literature emphasizes the potential of deep learning-based computer-aided diagnosis (CAD) [29, 23, 15, 21], e.g., demonstrating that a symbiosis of deep learning models with radiologist assessment yields the highest breast cancer detection performances [20]. However, training deep learning models on patient data poses a risk of leakage of sensitive person-specific information during and after training [23], as models have the capacity to memorise sufficient information to allow for high-fidelity image reconstruction [3, 13]. To avoid such leakage of private patient information, data needs to be protected during model training, in particular when the objective is to develop models to be used in clinical practice or shared among entities. Furthermore, international data protection regulations grant patients the right to request the removal of their information from data holders. For instance, point (b) of article 17(1) of the EU General Data Protection Regulation (GDPR) [9] stipulates that data subjects have a “right to be forgotten”. Given, for instance, the proven possibility of reconstructing training data given a model’s weights [3, 13], these rights can extend to the removal of patient-specific information from already trained deep learning models [28]. However, it is known to be difficult to “reliably” and “provably” remove patient information — present in only one or few specific training data points — from already trained model weights [28]. A generic and verifiable alternative is given by the removal of a patient’s data point from the training data and retraining of the respective model with the reminder of the dataset. This procedure is not only likely to have negative impacts on the performance of algorithms, but also emerges as a deterrence and risk for hospitals to adopt deep learning models, due to extensive economic, organisational, and environmental costs caused by retraining. Anticipating patient consent withdrawals, costly retraining can be avoided by demonstrating that deep learning model weights do not include personally identifiable information (PII) about any specific patient. To this end, a powerful technique to ensure privacy during model training is given by Differentially Private Stochastic Gradient Descent (DP-SGD)[1], which quantifiably reduces the effect each single training sample can have on the resulting model weights. Furthermore, privacy-preservation can also be achieved by diagnostic models exclusively trained on synthetic data, which is not (unambiguously) attributable to any specific patient but rather contains anonymous samples representing the essence of the dataset [12, 23]. The caveat of both DP-SGD and synthetic data strategies is, however, that they generally lead to a reduction in model performance, known as the privacy-utility trade-off. Investigating this trade-off in the realm of breast imaging, our core contributions are summarised as follows:

• We design and validate a transformer model, achieving promising performance as a backbone for privacy-preserving breast mass malignancy classification.

• We propose and validate a conditional generative adversarial network capable of differentiating between benign and malignant breast mass generation.

• We empirically quantify privacy-utility-tradeoffs in mass malignancy classification, assessing various differential privacy guarantees, and further combine and compare them with training on synthetic data.

2 Methods and Materials

Datasets and Preprocessing

We use the open-access Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM) dataset [16], which consists of 891 scanned film mammography cases with segmented masses with biopsy-proven malignancy status. After extracting mass images from craniocaudal view (CC) and mediolateral oblique (MLO) views, we follow the predefined per-patient train-test split [16], allocating 1296 mass images for training and 402 (245 benign, 157 malignant) mass images to testing. We further divided this training set randomly per-patient into a training (1104 mass images, 525 malignant) and a validation set (192 mass images, 102 malignant). As external test set, we further adopt the publicly available BCDR cohort [19], which comprises 1010 patients, totalling 1493 lesions (639 masses) with biopsy information from both digital mammograms (BCDR-DM) and film mammograms (BCDR-FM). Our final BCDR test set contains 1106 mass images extracted from CC and MLO views, 486 of which are malignant and 620 benign. To obtain mass patches, the lesion contour information is used to extract bounding boxes from the mammograms. We then create a square patch with a minimum size of 128x128 around this bounding box, ensuring a margin of 60 pixel in each direction. For classification, the mass patches are resized to pixel dimensions of 224x224 using inter-area interpolation, maintaining image ratios, and stacked to 3 channels. Models were trained on either a single 8GB NVIDIA RTX 2080 Super or 48GB RTX A6000 GPU using PyTorch and opacus [30] for DP-SGD.

Cancer Classification Transformer Model

Given its reported high performance on classifying the presence of a lesion in mammography patches [29] and its shifted window mechanism, allowing to effectively attend to shapes of varying sizes, we adopt a swin transformer (Swin-T) [17] as cancer classification model, to distinguish between benign and malignant masses. We inititalize ImageNet-pretrained [6] network weights and, after following the Swin-T hyperparameter setup [17] (stride, window size), we adjust the last fully-connected layer of the swin transformer reinitializing it with two output nodes each one outputting the logits for one of our respective classes (i.e., malignant or benign). We only set the parameters of the adjusted fully-connected layer as trainable and apply a learning rate of 1e-5. A weight decay of 1e-8 is used following the fine-tuning experiment described in [17]. Furthermore, an adamw optimizer, label smoothing of 0.1, and a batch size of 128 are used. During training, random horizontal and vertical flips are applied as data augmentation and a cross entropy loss is backpropagated. Training for 300 epochs, the model from the epoch with the lowest area under the precision-recall curve (AUPRC) on the validation set is selected for testing.

Malignancy-Conditioned Generative Adversarial Network

Going beyond unconditional mass synthesis in the literature [29, 2], we propose a malignancy conditioned generative adversarial network (MCGAN) to control the generation of either benign or malignant synthetic breast masses. In general, GANs consist of a generator (G) and a discriminator (D) network, which engage in a two-player zero-sum game, where G generates synthetic samples that D strives to distinguish from real ones [12]. We design G and D as deep convolutional neural networks [26] and, as shown in Fig. 1, integrate class-conditional information [22]. To this end, we extract the histopathology report’s biopsy information for each mass from the metadata, and convert it into a discrete malignancy label. Then, we transform this label into a multi-dimensional embedding vector before passing it through a fully-connected layer yielding a representation with the corresponding dimensionality to concatenate it to the generator input (100 dim noise vector) and to the discriminator input (128x128 input image). As D learns to associate class labels with patterns in the input images, it has to learn whether or not a given class corresponds to a given synthetic sample. Furthermore, as the discriminator loss is backpropagated into the generator, G is forced to synthesize samples corresponding to the provided class condition. This results in G learning a conditional distribution based on the value function

$$\min_{G}\max_{D}V(D,G)=\min_{G}\max_{D}[\mathbb{E}_{x\sim p_{\mathrm{data}}}[\log D(x|y)]+\mathbb{E}_{z\sim p_{z}}[\log(1-D(G(z|y)))]].$$

Optimizing the discriminator via binary cross-entropy [12], we define its loss in a class-conditional setup as

$$L_{D_{\mathrm{MCGAN}}}=-\mathbb{E}_{x\sim p_{\mathrm{data}}}[\log D(x|y)]+\mathbb{E}_{z\sim p_{z}}[\log(1-D(G(z|y)))].$$

We train our MCGAN on the CBIS-DDSM training data, applying random horizontal (p=0.5) and vertical (p=0.5) flipping as well as random cropping with resizing, where the resize scale ranges from 0.9 to 1.1 and aspect ratio from 0.95 to 1.1. We further include one-sided label smoothing [26] in a range of [0.7, 1.2]. Following [2], we employ a discriminator convolutional kernel size of 6 and a generator kernel size of 4. We observe that this reduces checkerboard artefacts as D’s field-of-view now requires G to create realistic transitions between the kernel-sized patches in the image. MCGAN is trained for 10k epochs with a batch size of 16. Based on the best quality-diversity tradeoff, we select the model from epoch 1.4k after qualitative visual assessment of generated samples .

Patient Privacy Preservation Framework

Privacy protection is an ethical norm and legal obligation, e.g. granting patients the right of their (retrospective) removal from databases [9]. Since (biomedical) deep learning models are vulnerable to information leakage, e.g. sensitive patient attributes [28, 3, 13], they can be affected by such (and future) regulations. However, privacy-preserving techniques can be integrated into deep learning frameworks and, to some extent, avoid compromising confidential data. For instance, (i) model training with DP-SGD [1] or (ii) training exclusively on synthetic data.

From a legal perspective, models trained on only synthetic data remain unaffected by patient consent withdrawal if “relatedness” between the data and the data subject cannot be established, or if “personal data has been rendered synthetic in such a manner that the data subject is no longer identifiable” [18] e.g., according to article 4(1) and recital 26 of the GDPR [9]. It is to be noted that in the “acceptable-risk” legal interpretation, a data subject’s re-identification risk is reduced to an “acceptable” level rather than fully eradicated [18]. Hence, this interpretation enables approaches such as synthetic data and/or Differential Privacy (DP) model training to be used as legally compliant privacy preservation methods despite not guaranteeing a “zero-risk” of patient re-identification.

DP is a mathematical framework that allows practitioners to provide (worst-case scenario) theoretical privacy guarantees for an individual sharing their data to train a deep learning model. Consider two databases (e.g., containing image-label pairs), we call them adjacent if they differ in a single data point, i.e., one image is present in one database but not in the other. Then, a randomised mechanism $\mathcal{M}\colon\mathcal{D}\rightarrow\mathcal{R}$ with domain $\mathcal{D}$ and range $\mathcal{R}$ is said to satisfy $(\varepsilon,\delta)$-differential privacy, if for any two adjacent databases $d,d^{\prime}\in\mathcal{D}$ and for any subset of outputs $S\subseteq\mathcal{R}$, $\Pr[\mathcal{M}(d)\in S]\leq e^{\varepsilon}\Pr[\mathcal{M}(d^{\prime})\in S]+\delta$ holds. $\varepsilon$ and $\delta$ bound a single data point’s influence on a model’s output (e.g. the models’ weights or predictions). Thus, the smaller the value of these parameters, the higher the model’s privacy and the harder it is for an attacker to retrieve information about any training data point. DP-SGD [1] is the DP variant of the well-known SGD algorithm, and facilitates the training of a model under DP conditions. In particular, a model trained under $(\varepsilon,\delta)$-DP is robust to post-processing, meaning only using its output for further computations also satisfies $(\varepsilon,\delta)$-DP. Moreover, the choice of these parameters is application-dependent and normative [5] and varies strongly across real-world deployments [7]. In the case of mammography, multiple lesions of the same patient are available in the datasets, i.e. one from the CC view and one from the MLO view. Therefore, to preserve the privacy of one patient it is necessary to protect all their data points (i.e. all images). In such a case, DP group privacy is used to estimate a patient’s DP privacy guarantee. However, for simplicity, in our subsequent experiments, we provide image-level privacy guarantees rather than per patient.

3 Experiments and Results

3.0.1 Synthetic Data Evaluation

Experimental Setup		Metrics
Dataset 1	Dataset 2	FIDImg $\downarrow$	FIDRad $\downarrow$	FRD $\downarrow$
SynMCGAN	RealDDSM	58.00$\pm$0.72	0.81$\pm$.013	18.12$\pm$1.01
RealDDSM	RealDDSM	29.25$\pm$0.82	0.31$\pm$.019	3.48$\pm$.352
SynMCGAN	SynMCGAN	20.90$\pm$0.16	0.32$\pm$.012	0.57$\pm$.094
RealDDSM	RealBCDR	156.43$\pm$14.3	3.88$\pm$.351	277.63$\pm$39.0

Figure 2: Qualitative and quantitative synthesis results: Images are randomly selected malignant and benign real (CBIS-DDSM [16]) and MCGAN-generated masses. ImageNet [6] and RadImageNet [25, 21] based FID [14] and FRD [24] scores are reported as mean $\pm$ standard deviation based on 3 subsets randomly sampled per patient (N${}_{\mathrm{real}}\approx 360$, N${}_{\mathrm{syn}}\approx 3240$). Row 4 indicates an BCDR-based[19] upper bound for comparison with synthetic data metrics in row 1.

Table 1: Results for within-domain (CBIS-DDSM [16]) and out-of-domain (BCDR [19]) breast cancer malignancy classification masses extracted from mammograms. Syn indicates 3k synthetic images being part of the fine-tuning training data, while SynPre represents pretraining all trainable model params with those 3k synthetic images (without DP guarantee), before fine-tuning the last two layers on real data with DP guarantee (RealFT). AUROC and AUPRC are reported as mean $\pm$ std based on 3 random seed runs. Best results in bold.

Experimental Setup			CBIS-DDSM [16]		BCDR [19]
Model	$\varepsilon$	$\delta$	AUROC $\uparrow$	AUPRC $\uparrow$	AUROC $\uparrow$	AUPRC $\uparrow$
SwinTReal	$\infty$	$\infty$	0.778$\pm$.001	0.85$\pm$.001	0.695$\pm$.002	0.726$\pm$.003
SwinTSyn	$\infty$	$\infty$	0.597$\pm$.011	0.696$\pm$.011	0.566$\pm$.064	0.602$\pm$.048
SwinTSynPre	$\infty$	$\infty$	0.639$\pm$.016	0.733$\pm$.001	0.622$\pm$.032	0.660$\pm$.017
SwinTReal	$1$	$1e^{-4}$	0.525$\pm$.043	0.640$\pm$.030	0.487$\pm$.020	0.549$\pm$.020
SwinTReal+Syn	$1$	$1e^{-4}$	0.553$\pm$.040	0.665$\pm$.025	0.521$\pm$.023	0.573$\pm$.024
SwinTSynPre+RealFT	$\infty$|$1$	$\infty$|$1e^{-4}$	0.661$\pm$.018	0.741$\pm$.007	0.637$\pm$.026	0.67$\pm$0013
SwinTReal	$6$	$1e^{-4}$	0.572$\pm$.031	0.679$\pm$.019	0.532$\pm$.031	0.579$\pm$.029
SwinTReal+Syn	$6$	$1e^{-4}$	0.617$\pm$.013	0.708$\pm$.015	0.609$\pm$.027	0.647$\pm$.024
SwinTSynPre+RealFT	$\infty$|$6$	$\infty$|$1e^{-4}$	0.677$\pm$.014	0.752$\pm$.009	0.647$\pm$.022	0.679$\pm$.009
SwinTReal	$12$	$1e^{-4}$	0.596$\pm$.023	0.702$\pm$.013	0.559$\pm$.033	0.600$\pm$.030
SwinTReal+Syn	$12$	$1e^{-4}$	0.624$\pm$.010	0.704$\pm$.012	0.625$\pm$.020	0.663$\pm$.012
SwinTSynPre+RealFT	$\infty$|$12$	$\infty$|$1e^{-4}$	0.688$\pm$.012	0.758$\pm$.011	0.654$\pm$.019	0.685$\pm$.007
SwinTReal	$20$	$1e^{-4}$	0.611$\pm$.018	0.715$\pm$.012	0.581$\pm$.028	0.618$\pm$.026
SwinTReal+Syn	$20$	$1e^{-4}$	0.630$\pm$.003	0.699$\pm$.008	0.641$\pm$.018	0.685$\pm$.012
SwinTSynPre+RealFT	$\infty$|$20$	$\infty$|$1e^{-4}$	0.697$\pm$.012	0.763$\pm$.012	0.659$\pm$.017	0.689$\pm$.006
SwinTReal	$60$	$1e^{-4}$	0.622$\pm$.014	0.721$\pm$.110	0.605$\pm$.019	0.640$\pm$.017
SwinTReal+Syn	$60$	$1e^{-4}$	0.629$\pm$.002	0.694$\pm$.005	0.650$\pm$.013	0.696$\pm$.007
SwinTSynPre+RealFT	$\infty$|$60$	$\infty$|$1e^{-4}$	0.712$\pm$.013	0.776$\pm$.013	0.671$\pm$.014	0.697$\pm$.004

Qualitatively assessing the synthetic images in Fig. 2, it is not readily possible to distinguish synthetic from real masses in terms of image fidelity or diversity. We note the absence of clear visual indicators to distinguish between malignant and benign images for both real and synthetic images. This is in line with the difficulty of determining the malignancy of a mammographic lesion shown by high clinical error rates and inter-observer variability [8]. However, results for training our malignancy classification model on only synthetic data (see Syn and SynPre in Table 1) show that the synthetic data captures the conditional distribution effectively generating either malignant or benign masses. Both, vanilla ImageNet-based Fréchet Inception Distance (FID) [14, 6] and radiology domain-specific RadImageNet-based FID [25, 21], concur that the synthetic data (FID_Img=58$\pm$.72) is substantially closer to the real CBIS-DDSM [16] distribution compared to BCDR [19] (FID_Img=156.43$\pm$1.43). This is even more pronounced when comparing the variation of extracted radiomics features for CBIS-DDSM to synthetic (FRD=18.12) and BCDR (FRD=277.63) images using the Fréchet Radiomics Distance (FRD) [24]. While this indicates desirable synthetic data fidelity, we also observe good diversity. The latter is shown by comparing subsets of the same datasets with each other, where the variation within the synthetic data (e.g., FID_Rad=0.32$\pm$.12) closely resembles the variation within the real CBIS-DDSM dataset (e.g., FID_Rad=0.31$\pm$.19). Notwithstanding less variation in radiomics imaging biomarkers within the synthetic data (FRD_Syn=0.57 vs. FRD_Real=3.48), this overall points to a valid coverage of the distribution and an absence of mode collapse.

3.0.2 Mass Malignancy Classification

As shown in Table 1, we conduct experiments with and without formal privacy guarantees. For scenarios where a formal privacy guarantee is not strictly required and, thus, synthetic data suffices as privacy mechanism, we compare the results of training SwinT on synthetic data (Syn) and on real data (Real) with DP-SGD. Kaissis et al. [15] defined $\varepsilon=6$ as suitable privacy budget for their medical imaging dataset. Compared to DP-SGD with $\varepsilon=6$, synthetic data achieves better AUPRCs for within-domain tests on CBIS-DDSM (SwinT_Syn=0.696 vs SwinT_Real(ε=6)=0.679) and is on par for out-of-domain (ood) tests on BCDR (SwinT_Syn=0.602 vs SwinT_Real(ε=6)=0.600). However, training all SwinT layers using synthetic data (SynPre), achieves substantially better performance only approximated by DP results for $\varepsilon=60$ for within-domain (SwinT_SynPre=0.733 vs SwinT_Real(ε=60)=0.721) and ood (SwinT_SynPre =0.66 vs SwinT_Real(ε=60)=0.64) tests. Further fine-tuning SwinT_SynPre on real data using DP-SGD results in additional improvement across all privacy parameters for within-domain and ood testing. For instance, training SwinT_{SynPre+RealFT} with $\varepsilon=1$ results in an AUPRC of 0.74 and 0.67 for CBIS-DDSM and BCDR, respectively. To assess scenarios where a formal guarantee is required, we further compare DP-SGD training of SwinT on real data (Real) with DP-SGD training on a mix of real and synthetic data (Real+Syn). To this end, our experiments show that such synthetic data augmentation can improve the privacy-utility tradeoff. This is exemplified by SwinT_{Real+Syn(ε=6)} accomplishing an AUPRC of 0.708 within-domain and 0.647 ood, while SwinT_Real(ε=6) achieved 0.679 and 0.579, respectively. We further observe the trend that stricter privacy budgets (i.e., smaller $\varepsilon$) can be associated with more added performance of synthetic data as additional classification model training data.

4 Discussion and Conclusion

We introduce a privacy preservation framework based on differential privacy (DP) and synthetic data and apply it to the diagnostic task of classifying the malignancy of breast masses extracted from screening mammograms. We further propose, train, and evaluate a malignancy-conditioned generative adversarial network to generate a dataset of benign and malignant synthetic breast masses. Next, we train a swin transformer model on mass malignancy classification and assess, compare and combine training under DP and training on synthetic data. This analysis revealed that when training with DP, synthetic data augmentation can notably improve classification performance for within-domain and out-of-domain test cases. Apart from that, we show, across privacy mechanisms and across domains, that the performance of models pretrained on synthetic data can be further improved by DP fine-tuning on real data.

This finding is particularly important considering that synthetic data, if not directly attributable to any specific patient, can become a valid, legally compliant alternative to strict DP guarantees in clinical practice. Consequently, it is to be further investigated where and when deterministic mechanisms without formal DP guarantees can suffice to shield against different privacy attacks [4]. In particular, we motivate future work to analyse the extent to which the inherent properties of synthetic data generation algorithms can provide empirical protection against attacks. A methodological alternative to our approach is to assess privacy-utility tradeoffs when training the generative model itself using DP-SGD [10, 23], resulting in formal privacy guarantees of the generated synthetic datasets. Thus, a further avenue to explore then lies within the question whether randomness inherent in randomised data synthesis algorithms (e.g., based on the noise in diffusion models [27] or GANs [12]) can be used to amplify the privacy of the DP versions of such synthesis algorithms, thereby potentially further enhancing privacy-utility tradeoffs. To this end, our study constitutes a crucial first step leading towards the clinical adoption of diagnostic deep learning models, enabling practical privacy-utility tradeoffs all while anticipating respective legal obligations and clinical requirements.

4.0.1 Acknowledgements

This study has received funding from the European Union’s Horizon research and innovation programme under grant agreement No 952103 (EuCanImage) and No 101057699 (RadioVal). It was further partially supported by the project FUTURE-ES (PID2021-126724OB-I00) from the Ministry of Science and Innovation of Spain. RO acknowledges a research stay grant from the Helmholtz Information and Data Science Academy (HIDA).

4.0.2

The authors have no competing interests to declare that are relevant to the content of this article.