A Survey of Privacy Attacks in Machine Learning

The rest of the article is organized as follows: Section 2 introduces some basic concepts related to machine learning that are relevant to the implementation of the attacks, which are presented in Section 6. The threat model is presented in Section 3 and the taxonomy of the attacks and their definition are the focus of Section 4. In Section 5, we present the causes of machine learning leaks that are known or have been investigated so far. An overview of the proposed defences per attack type is the focus of Section 7. Finally, Section 8 contains a discussion on the current and future research directions and Section 9 offers concluding remarks.

At a very high level, ML is traditionally split into three major areas: supervised, unsupervised, and reinforcement learning. Each of these areas has its own subdivisions. Over the years, new categories have emerged to capture types of learning that cannot easily fit under these three areas such as semi-supervised and self-supervised learning, or other ways to categorize models such as generative and discriminative ones.

From a system architecture point of view, we view the learning process as either a centralized or a distributed one. The main criterion behind this categorization is whether the data and the model are collocated or not.

Training of supervised ML models usually follows the Empirical Risk Minimization (ERM) approach [126], where the objective is to find the parameters \(\mathbf {\theta }^*\) that minimize the risk or objective function, which is calculated as an average over the training dataset:where \(l(\cdot)\) is a loss function, e.g., cross entropy loss, and m is the number of data points in the dataset \(\mathcal {D}\).

The idea behind ERM is that the training dataset is a subset drawn from the unknown true data distribution for the learning task. Since we have no knowledge of the true data distribution, we cannot minimize the true objective function, but instead we can minimize the estimated objective over the data samples that we have. In some cases, a regularization term is added to the objective function to reduce overfitting and stabilize the training process.

Membership inference tries to determine whether an input sample \(\mathbf {x}\) was used as part of the training set \(\mathcal {D}\). This is the most popular category of attacks and was first introduced by Shokri et al. [116]. The attack only assumes knowledge of the model’s output prediction vector (black-box) and was carried out against supervised machine learning models. White-box attacks in this category are also a threat, especially in a collaborative setting, where an adversary can mount both passive and active attacks. If there is access to the model parameters and gradients, then this allows for more effective white-box membership inference attacks in terms of accuracy [90].

Apart from supervised models, generative models such as GANs and VAEs are also susceptible to membership inference attacks [16, 39, 44]. The goal of the attack, in this case, is to retrieve information about the training data using varying degrees of knowledge of the data generating components.

Finally, these types of attacks can be viewed from a different perspective, that of the data owner. In such a scenario, the owner of the data may have the ability to audit black-box models to see if the data have been used without authorization [46, 118].

Reconstruction attacks try to recreate one or more training samples and/or their respective training labels. The reconstruction can be partial or full. Previous works have also used the terms attribute inference or model inversion to describe attacks that, given output labels and partial knowledge of some features, try to recover sensitive features or the full data sample. For the purpose of this survey, all these attacks are considered as part of the larger set of reconstruction attacks. The term attribute inference has been used in other parts of the privacy related literature to describe attacks that infer sensitive “attributes” of a targeted user by leveraging publicly accessible data [31, 53]. These attacks are not part of this review as they are mounted against the individual’s data directly and not against ML models.

A major distinction between the works of this category is between those that create an actual reconstruction of the data [41, 134, 138, 145, 148] and the ones that create class representatives or probable values of sensitive features that do not necessarily belong to the training dataset [28, 43, 47, 109, 138]. In classification models, the latter case is limited to scenarios where classes are made up of one type of object, e.g., faces of the same person. While this limits the applicability of the attack, it can still be an interesting scenario in some cases.

The ability to extract dataset properties that were not explicitly encoded as features or were not correlated to the learning task is called property inference. An example of property inference is the extraction of information about the ratio of women and men in a patient dataset when this information was not an encoded attribute or a label of the dataset. Or having a neural network that performs gender classification and can be used to infer if people in the training dataset wear glasses or not. In some settings, this type of leak can have privacy implications. These types of properties can also be used to get more insight about the training data, which can lead to adversaries using this information to create similar models [3] or even have security implications when the learned property can be used to detect vulnerabilities of a system [29].

Property inference aims to extract information that was learned from the model unintentionally and that is not related to the training task. Even well generalized models may learn properties that are relevant to the whole input data distribution and sometimes this is unavoidable or even necessary for the learning process. What is more interesting from an adversarial perspective, are properties that may be inferred from a specific subset of training data, or eventually about a specific individual.

Property inference attacks so far target either dataset-wide properties [3, 29, 78, 117] or the emergence of properties within a batch of data [84]. The latter attack was performed on the collaborative training of a model.

Model extraction is a class of black-box attacks where the adversary tries to extract information and potentially fully reconstruct a model by creating a substitute model \(\hat{f}\) that behaves very similarly to the model under attack f. There are two main goals for the substitute models. First, to create models that match the accuracy of the target model f in a test set that is drawn from the input data distribution and related to the learning task [63, 86, 98, 124]. Second, to create a substitute model \(\hat{f}\) that matches f at a set of input points that are not necessarily related to the learning task [21, 50, 56, 124]. Jagielski et al. [50] referred to the former attack as task accuracy extraction and to the latter as fidelity extraction. In task accuracy extraction, the adversary is interested in creating a substitute that learns the same task as the target model equally well or better. In the latter case, the adversary aims to create a substitute that replicates the decision boundary of f as faithfully as possible. This type of attack can be later used as a stepping stone before launching other types of attacks such as adversarial attacks [56, 103] or membership inference attacks [90]. In both cases, it is assumed that the adversary wants to be as efficient as possible, i.e., to use as few queries as possible. Knowledge of the target model architecture is assumed in some works, but it is not strictly necessary if the adversary selects a substitute model that has the same or higher complexity than the model under attack [56, 63, 98].

Apart from creating substitute models, there are also approaches that focus on recovering information from the target model, such as hyper-parameters in the objective function [131] or information about various neural network architectural properties such as activation types, optimisation algorithm, number of layers, and so on [97].

One of the conditions that has been shown to improve the accuracy of membership inference is the poor generalization of the model. The connection between overfitting and black-box membership inference was initially investigated by Shokri et al. [116]. This paper was the first to examine membership inference attacks on neural networks. The authors measured the effect of overfitting on the attack accuracy by training models in different MLaaS platforms using the same dataset. The authors showed experimentally that overfitting can lead to privacy leakage but also noted that it is not the only condition, since some models that had lower generalization error where more prone to membership leaks. The effect of overfitting was later corroborated formally by Yeom et al. [140]. The authors defined membership advantage as a measure of how well an attacker can distinguish whether a data sample belongs to the training set or not, given access to the model. They proved that the membership advantage is proportional to the generalization error of the model and that overfitting is a sufficient condition for performing membership inference attacks, but not a necessary one. Additionally, Long et al. [74] showed that even in well-generalized models, it is possible to perform membership inference for a subset of the training data, which they named vulnerable records. This was also corrobaorated in Reference [11]. In addition, Carlini et al. showed that larger language models are more prone to memorization than smaller models [12].

Other factors, such as the model architecture, model type, and dataset structure, affect the attack accuracy. Similar to Reference [116] but in the white-box setting, Nasr et al. [90] showed that two models with the same generalization error resulted to different degrees of leakage. More specifically, the most complex model in terms of number of parameters exhibited higher attack accuracy, showing that model complexity is also an important factor. In a white-box setting also, Leino and Fredrikson [66] showed that the learned features of the target model capture differences between the training data distribution and the general population. These differences can be used to distinguish members of the training set even when the target model generalizes well.

Truex et al. [125] ran different types of experiments to measure the significance of the model type as well as the the number of classes present in the dataset. They found that certain model types such as Naive Bayes are less susceptible to membership inference attacks than decision trees or neural networks. They also showed that as the number of classes in the dataset increases, so does the potential of membership leaks. This finding agrees with the results in Reference [116].

Securing machine learning models against adversarial attacks can also have an adverse effect on the model’s privacy as shown by Song et al. [120]. Current state-of-the-art proposals for robust model training, such as projective gradient descent (PGD) adversarial training [77], increase the model’s susceptibility to membership inference attacks. This is not unexpected, since robust training methods (both empirical and provable defenses) tend to increase the generalization error. As previously discussed, the generalization error is related to the success of the attack. Furthermore, the authors of Reference [120] argue that robust training may lead to increased model sensitivity to the training data, which can also affect membership inference.

The generalization error is easily measurable in supervised learning under the assumption that the test data can capture the nuances of the real data distribution. In generative models and specifically in GANs this is not the case, hence the notion of overfitting is not directly applicable. All three papers that deal with membership inference attacks against GANs mention overfitting as an important factor behind successful attacks [16, 39, 44]. In this case, overfitting means that the generator has memorized and replays part of the training data. This is further corroborated in the study in Reference [16], where their attacks are shown to be less successful as the training data size increases.

Regarding reconstruction attacks, Yeom et al. [140] showed that a higher generalization error can lead to a higher probability to infer data attributes but also that the influence of the target feature on the model is an important factor. However, the authors assumed that the adversary has knowledge of the prior distribution of the target features and labels. Using weaker assumptions about the adversary’s knowledge, Zhang et al. [145] showed theoretically and experimentally that a model that has high predictive power is more susceptible to reconstruction attacks.

Property inference is possible even with well-generalized models [29, 84] so overfitting does not seem to be a cause of property inference attacks. Unfortunately, regarding property inference attacks, we have less information about what makes them possible and under which circumstances they appear to be effective. This is an interesting avenue for future research, both from a theoretical and an empirical point of view.

While overfitting increases the success of black-box membership inference attacks, the exact opposite holds for model extraction attacks. It is possible to steal model parameters when the models under attack have 98% or higher accuracy in the test set [97]. Also models with a higher generalization error are harder to steal, probably due to the fact that they may have memorized samples that are not part of the attacker’s dataset [73]. Another factor that may affect model extraction success is the dataset used for training. Higher number of classes may lead to worse attack performance [73].

In the centralized learning setting, the main assumption is that models and data are collocated during the training phase. The next subsection introduces a common design approach that is used by multiple papers, namely, the use of shadow models or shadow training. The rest of the subsections are dedicated to the different attack types and introduce the assumptions, common elements, as well as differences of the reviewed papers.

In the federated learning setting, multiple devices acquire access to the global model that is trained from data that belong to different end users. Furthermore, the parameter server has access to the model updates of each participant either in the form of model parameters or that of loss gradients. In split learning settings, the central server also gains access to the outputs of each participant’s intermediate neural network layers. This type of information can be used to mount different types of attacks by actors that are either residing in a central position or even by individual participants. The following subsection presents the types of attacks in distributed settings, as well as their common elements, differences, and assumptions.

To summarize the attacks proposed against machine learning privacy, Table 1 presents the 47 papers analyzed in terms of adversarial knowledge, model under attack, attack type, and timing of the attack.

In terms of model types, 86.8% of the papers dealt with attacks against neural networks, with decision trees and linear models being the second most popular models to attack at 10.6% (some papers covered attacks against multiple model types). The concept of neural networks groups together both shallow and deep models, as well as multiple architectures, such as convolutional neural networks, recurrent neural networks, while under SVMs, we group together both linear and nonlinear versions.

The most popular attack type is membership inference (41.5% of the papers) with reconstruction attacks the second most popular (30.2% of the papers) and model extraction in the third place (28.3%). The majority of the proposed attacks are performed during the inference phase (86.8%). Attacks during training are mainly on distributed forms of learning or related to poisoning. Black-box and white-box attacks were studied in 71.7% and 43.4% of the papers, respectively (some papers covered both settings). In the white-box category, we also include partial white-box attacks.

The focus on neural networks in the existing literature as well as the focus on supervised learning is also apparent in Figure 4. The figure depicts types of machine learning algorithms versus the types of attacks that have been studied so far based on the existing literature. The list of algorithms is indicative and not exhaustive, but it contains the most popular ones in terms of research and deployment in real-world systems. Algorithms such as random forests [10] or gradient boosting trees [17, 60] have received little to no focus and the same holds for whole areas of machine learning such as reinforcement learning.

Another dimension that is interesting to analyze is the types of learning tasks that have been the target of attacks so far. Figure 5 presents information about the number of papers in relation to the learning task and the attack type. By learning task, we refer to the task in which the target model is initially trained. As the figure clearly shows, the majority of the attacks are on models that were trained for classification tasks, both binary and multiclass. This is the case across all four attack types.

While there is a diverse set of reviewed papers, it is possible to discern some high-level patterns in the proposed attacking techniques. Figure 6 shows the number of papers in relation to the attacking technique and attack type. Most notably, 10 papers used shadow training mainly for membership and property inference attacks. Active learning was quite popular in model extraction attacks and was proposed by six papers. Generative models (mostly GANs) were used in five papers across all attack types and another three papers used gradient matching techniques. It should be noted here that the “Learning” technique includes a number of different approaches, spanning from using model parameters and gradients as inputs to classifiers [84, 89] to using input-output queries for substitute model creation [21, 50, 98] and learning classifiers from language models for reconstruction [101] and property inference [117]. In “Threshold”-based attacks, we categorized the attacks proposed in References [140] and [108] and subsequent papers that used them for membership and property inference.

Some attacks may be applicable to multiple learning tasks and datasets, however, this is not the case universally. Dataset size, number of classes, and features might also be factors for the success of certain attacks, especially since most of them are empirical. Table 2 is a summary of the datasets used in all attack papers along with the data types of their features, the learning task they were used for, and the dataset size. The datasets were used during the training of the target models and in some cases as auxiliary information during the attacks. The table contains 56 unique datasets used across 53 papers, an indication of the variation of different approaches.

This high variation is both a blessing and a curse. On the one hand, it is highly desirable to use multiple types of datasets to test different hypotheses and the majority of the reviewed research follows that approach. On the other hand, these many options make it harder to compare methods. As it is evident from Table 2, some of the datasets are quite popular. MNIST, CIFAR-10, CIFAR-100, and UCI Adult have been used by more than nine papers, while 29 datasets have been used by only one paper.

The number of model parameters varies based on the model, task and datasets used in the experiments. As it can be seen in Table 2, most datasets are not extremely large, hence the models under attack are not extremely large. Given that most papers deal with neural networks, this might indicate that most attacks focused on smaller datasets and models, which might not be representative of realistic scenarios. However, privacy attacks do not necessarily have to target large models with extreme amounts of data; and neural networks, however popular, are not necessarily the most used models in the “real world.”

The most prominent defense against membership inference attacks is Differential Privacy (DP), which provides a guarantee on the impact that single data records have on the output of an algorithm or a model. However, other defenses have been tested empirically and are also presented in the following subsections.

Reconstruction attacks often require access to the loss gradients during training. Most of the defences against reconstruction attacks propose techniques that affect the information retrieved from these gradients. Setting all loss gradients that are below a certain threshold to zero was proposed as a defence against reconstruction attacks in deep learning. This technique proved quite effective with as little as 20% of the gradients set to zero and with negligible effects on model performance [148]. However, performing quantization or using half-precision floating points for neural network weights did not seem to deter the attacks in References [11] and [148], respectively.

Differential privacy is designed to provide privacy guarantees in membership inference attack scenarios and it does not seem to offer protection against property inference attacks [3]. In addition to DP, Melis et al. [84] explored other defenses against property inference attacks. Regularization (dropout) had an adverse effect and actually made the attacks stronger. Since the attacks in Reference [84] were performed in a collaborative setting, the authors tested the proposal in Reference [115], which is to share fewer gradients between training participants. Although sharing less information made the attacks less effective, it did not alleviate them completely.

Model extraction attacks usually require that the attacker performs a number of queries on the target model. The goal of the proposed defenses so far has been the detection of these queries. This contrasts with the previously presented defences that mainly try to prevent attacks.

Attacks on machine learning privacy have been increasingly brought to light. However, we are still at an exploratory stage. For some attacks such as membership inference, we know that there is a connection to overfitting and memorization but there are also indications that some data points are easier to infer than others. Currently, we lack a deeper understanding about the rest of the inference attacks, especially property inference and model extraction attacks. Potential future work may be done, using a data-centric approach, on why some data points may be easier to infer, and which are their properties.

As much as we need answers about why leaks happen at a theoretical level, we also need to know how well privacy attacks work on real deployed systems. Adversarial attacks on realistic systems bring to light the issue of additional constraints that need to be in place for the attacks to work. For example, when creating glasses to fool a face recognition system, Sharif et al. [114] had to pose constraints that had to do with physical realizations, e.g., that the color of the glasses should be printable. In privacy-related attacks, the most realistic cases come from the model extraction area, where attacks against MLaaS systems have been demonstrated in multiple papers. For the majority of other attacks, it is certainly an open question of how well they would perform on deployed models and what kind of additional requirements would need to be in place for them to succeed. Potential future work may be done by exploring attacks in real-world implementations and deployments, where the assumptions about the access to the training dataset are a challenge.

Beyond expanding the focus on different learning tasks, there is the question of datasets. The impact of datasets on the attack success has been demonstrated by several papers. Yet, we currently lack a common approach as to which datasets are best suited to evaluate privacy attacks, or what constitutes the minimum requirements for a successful attack. Several questions are worth considering: Do we need standardized datasets? If yes, then how do we go about and create them? Are all data worth protecting? If some are more interesting than others, then is it not a good idea to test attacks beyond popular image datasets? A potential future work would be the standardization of datasets and testing environments. This may be an avenue that would benefit researchers to advance the state of the art, as well as engineers that deploy models in production environments. Part of this future direction should also include guidelines for evaluation and deployment with respect to potential privacy leaks.

The main research focus up to now has been supervised learning. Even within supervised learning, there are areas and learning tasks that have been largely unexplored. Information leakage in newer model architectures such as attention networks and graph neural networks just recently started to attract some focus. In addition there are very few attacks reported on popular algorithms such as random forests or gradient boosting trees despite their wide application and this is research area that deserves more attention and future work. In unsupervised and semi-supervised learning, the focus is mainly on generative models, and only just recently, papers started exploring areas such as representation learning and language models. Some attacks on image classifiers do not transfer that well to natural language processing tasks [46], while others do but may require different sets of assumptions and design considerations [101]. Figure 4 shows which type of attacks were still not tried in which models or algorithms. All these are potential future work directions.

Finally, as we strive to understand the privacy implications of machine learning, we also realize that several research areas are connected and affect each other. We know, for instance, that adversarial training (using adversarial samples to make the model more robust) affects membership inference [119] and that model censoring can still leak private attributes [119]. Property inference attacks can deduce properties of the training dataset that were not specifically encoded or were not necessarily correlated to the learning task. This can be understood as a form of bias detection, which means that relevant literature in the area of model fairness should be reviewed as potentially complementary. Furthermore, while deep learning models are considered black-boxes in terms of explainability, future work that sheds light on what type of data make neurons activate [94, 141] can be relevant to discovering information about the target’s training dataset and can therefore lead to privacy leaks. All these are examples of potential inter-dependencies between different areas of machine learning research. Therefore, a better understanding of privacy attacks calls for an interdisciplinary approach.