Gender Bias in Natural Language Processing and Computer Vision: A Comparative Survey

Task-agnostic bias metrics target models that are pre-trained to be used as input representations in later tasks, therefore these metrics measure semantic bias (cf. Table 1). As these metrics are applied to the trained model they are independent of the domain and therefore of the dataset. These models are either LLMs or their predecessor word embeddings. Some of the methodology that was first developed to show how pre-trained word embeddings capture social biases were later adapted for LLMs [53], however, other methods were tailored to the LLMs’ context-dependent structure and training objective as language models. An overview of the methods discussed in this section can be found in Table 2.

Embedding Association Tests. One of the most commonly used frameworks for gender bias detection in NLP applications is the Word Embedding Association Test (WEAT). The test was adapted by Caliskan et al. [26] from the Implicit Association Test used in psychological research [52]. The WEAT measures associations between identity terms that express gender, such as he, she, and so on, and positive or negative terms, or terms relating to fields with a stereotyped gender-connotation, such as family life or natural sciences. The metric used here is the distance between the terms’ vector representations. While the WEAT has been praised for drawing on literature outside of NLP and thereby presenting an inter-disciplinary approach that grounds word embedding associations in human cognition [17], it has also been criticised for over-estimating bias [45].

With the emergence of LLMs, the WEAT was further adapted, as word representations in LLMs are not singular, like in traditional word embedding models, but are dependent on sentence context. May et al. [85] developed the Sentence Embedding Association Test (SEAT) and created “semantically bleached,” i.e., very simple, sentence templates into which the target and attribute terms were embedded to extract the respective vector representations of the words. The authors tested their methodology on a variety of LLMs, but found discrepancies in the results, leading them to question whether the concepts tested (e.g., gender or pleasantness) can be represented within simple sentences and their association measured using cosine similarity.

Avoiding the problem of sentence templates, Guo and Caliskan [53] also adapted the WEAT for use in LLMs. For their so-called Contextualized Embedding Association Test (CEAT), they extracted 10,000 sentences containing stimuli (target/attribute words) from a Reddit Corpus, compute the WEATs to obtain effect sizes for all pairings and then use a random effects model, which is used in meta analysis, to analyze the distribution of effect sizes. They found presence of all tested bias categories in all of the tested LLMs (GPT, GPT-2, BERT, ELMo) but also found some negative results, indicating that “some WEAT stimuli tend to occur in stereotype-incongruent contexts more frequently” [53].

Increased Log Probability Bias Score (ILPBS) . Kurita et al. [71] presented a method of measuring associations between identity terms and stereotypical terms within masked language models, such as BERT. The log probability bias score measures word likelihood in varying contexts. Similar to May et al. [85], they also created templates. However, instead of using semantically bleached contexts, their templates contain both an identity term and a stereotyped attribute term. The difference in association between counterfactual identity terms in sentences with the same attribute using an LLM is measured to create the score. Kurita et al. [71] found statistically significant bias scores where their adaptation of WEAT did not provide significant results. The authors interpret this as proof that gender bias assessment methods used on standard word embeddings cannot be simply translated to LLMs.

Discovery of Correlations (DisCo) . Webster et al. [137] took a slightly different template-based approach. Instead of measuring the associations between pre-defined terms, their method aimed to discover terms correlated with gender (DisCo). They created two template variants for DisCo, one that uses first names, e.g., “[NAME] studied [BLANK] at college,” and another that uses nouns that contain gender information, e.g., “The [NOUN] likes to [BLANK]” [137]. The model was then asked to fill in the blank slot. The researchers used the \(\chi ^2\) measure to see whether the three most likely proposals were significantly correlated with the associated gender of [PERSON] or [NAME], i.e., whether the proposed fill was gender-dependent, indicating model bias. Using DisCo, the researchers found that in both BERT and ALBERT, first names are more likely to generate fills with what they termed gendered correlations, than gendered nouns. Furthermore they showed that LLMs with similar accuracy do not necessarily show the same gendered correlations.

ABC Stereotype Model/ Sensitivity Test (SeT) . Another common criticism of research on bias and fairness in NLP is that techniques are not sufficiently grounded in theory from outside of the field, such as psychology, sociology, feminist theory [17]. Cao et al. [28] conducted a study asking participants to report stereotypes held by the general population and based their research on Koch et al.’s [68] Agency Beliefs Communion (ABC) stereotype model from social psychology theory. Cao et al. [28] also presented their own methodology for measuring stereotyping in LLMs: the SeT. The SeT measures how much model weights would need to change to arrive at predicting an anti-stereotypical trait for a given group, e.g., “Men are kind.” Compared to both CEAT [53] and the Log Probability Bias Score [71], the SeT showed better alignment with the ways in which humans tend to stereotype. However, overall human and model judgements only showed moderate correlation.

Open-ended Language Generation. Bias can also be measured through open-ended language generation. Sheng et al. [114] created sentence templates with placeholders for identity terms in contexts related to respect for a person and occupations. They used two measures to assess bias within the generated sentence completions by the LLM: sentiment as well as regard. Nozza et al. [92] also create templates that contain identity terms in different contexts, and they additionally presented the HONEST score, which uses the HurtLex lexicon of harmful language [9], to measure how often a language model’s top candidates for completing a sentence contain toxic language. They applied the HONEST score to BERT and GPT-2 models in six languages and found, for example, that sentence templates containing a female subject were completed with a reference to promiscuity 9% of the time. Nozza et al. [93] extended this research to LGBTQ+-related identity terms and measured the HONEST score as well as toxicity on the sentence level using the Perspective API.² They found that the completed sentences by the LLMs queried are classified to be harmful 13% of the time. Furthermore, Dhamala et al. [41] created the BOLD measures and dataset, which use sentences from Wikipedia that contain mentions of protected groups to measure bias in open-ended language generation. Akyürek et al. [3] took a critical perspective on using open-ended language generation as a measure for bias. They demonstrated that bias measures are highly dependant on experimental design, including factors like model parameters, which can influence whether or not bias reaches a harmfulness threshold.

Challenge Datasets. Measuring gender bias in NLP overall is dependant on datasets that contain identity terms for which different behaviours are measured. These challenge datasets are designed specifically to assess shortcomings with respect to a certain (social) variable. Challenge datasets are mentioned under several different names throughout the literature. Blodgett et al. [18] referred to “benchmark datasets,” thereby drawing the connection to performance-measuring benchmarks. Stanczak and Augenstein [119] called them “probing datasets” while Sun et al. [124] referred to “Gender Bias Evaluation Test sets.” Bowman [20] specifically mentioned adversarial datasets, for which annotators were asked to create cases that make a model fail.

Two challenge datasets to assess social biases in LLMs are CrowS-Pairs [91] and StereoSet [90]. CrowS-Pairs consists of minimal sentence pairs, one of which contains a stereotype and one which does not. By measuring the likelihood that a given language model gives to either sentence, social biases can be assessed. StereoSet is slightly more comprehensive; it features both intra- and inter-sentence settings. In the intra-sentence setting, sentences have a gap and multiple words to fill them: a stereotypical, a non-stereotypical and an unrelated filler. Measuring the likelihood of each of these fillers can provide indication of model bias. The inter-sentence setting is similarly structured, only here the likelihood of three possible sentence continuations is being measured. Despite being more comprehensive, the authors of CrowS-Pairs noted that the StereoSet has a lower annotator validation rate than CrowS-Pairs [91].

In this section, we present research on detecting gender bias in specific NLP tasks. While task-agnostic metrics are mostly architecture-dependent (e.g., suitable for masked and/or causal language models), task-specific metrics are less tied to a specific architecture, because different model architectures can be used for the same task. Instead, methodologies are often dependent on challenge datasets, which allows researchers to test model performance with regard to a protected variable, gender in our case. Therefore, in addition to the discussion of several works on task-specific gender bias, we provide a non-exhaustive overview of these datasets in Table 3.

Coreference Resolution. Challenge datasets for detecting gender bias exist for a broad variety of downstream NLP applications. For example, the WinoBias dataset [141] targets pronoun resolution in pro- and anti-stereotypical sentences, such as “The physician hired the secretary because he was highly recommended” [141]. Similar datasets, which also target binary gender bias in coreference resolution, are WinoGender [100] and GAP [136]. Cao and Daumé [27] then developed the GICoref dataset, which contains challenging coreference cases with non-binary and neopronouns, as well as gender-fluid cases, in which pronoun use changes while still referring to the same person.

Occupation Classification. De-Arteaga et al. [36] filtered almost 400,000 professional biographies from the Common Crawl Corpus and used this dataset to assess gender bias in occupation classification. They measured gender bias using the difference in true positive rate for male and female biographies per occupation in two settings: (1) gender markers contained in the biographies left as is or (2) they were “scrubbed.” De-Arteaga et al. [36] found a significant gender gap that was correlated with statistics of gender participation in the workforce. When “scrubbing” gender information, the gender gap was reduced, but the accuracy of the classifier remained stable.

Sentiment Analysis. One of the first to study biases in sentiment analysis (SA) systems were Kiritchenko and Mohammad [67]. They created the Equity Evaluation Corpus (EEC) that consists of sentences designed to target race and gender biases within an SA system. Using this corpus to evaluate 219 openly available SA systems, the authors found that around three quarters of the systems consistently attribute higher sentiment intensity to identity terms related to historically disadvantaged protected groups, such as women and Black people [67]. Based on this approach, Bhaskaran and Bhallamudi [14] created another EEC that is designed to expose gendered occupational stereotypes in SA systems. They found differing sentiment scores for male and female identity terms as well as more negative sentiment toward lower-earning versus higher-earning jobs. Addressing limitations posed by handcrafted templates, Asyrofi et al. [5] created BiasFinder, a system that automatically generates templates that differ in identity terms of the same protected group, and for which different transformer-based SA systems predict differing sentiment. They call these sentence templates “bias-uncovering test sets” (BTC). On average, their SA systems find around 8,000 of these in an IMDB movie review dataset [79] and 24,000 in the Twitter Sentiment140 dataset [1].

Machine Translation. Another application of NLP for which bias is measured is machine translation (MT). Gender bias in MT becomes evident, for example, when translating from a non-gender marking language to a gender-marking language, in which the choice of grammatical gender for an originally gender-neutral word is based on stereotypes or societal gender roles. For example, the phrase “The doctor and the nurse” would be translated into German as “Der Arzt_masc und die Krankenschwester_fem.” While it could be argued that this translation simply reflects numbers of male and female participation in the respective professions, Prates et al. [96] established that Google Translate, for instance, has a tendency to create male-default translations and to over-amplify men’s participation in STEM fields. In a similar study, Cho et al. [31] showed a similar male skew for gender-neutral pronoun translation from Korean to English. Besides using occupation words, they also demonstrated this skew for gender-neutral pronoun translation in formal/informal contexts, and contexts containing words that carry positive or negative sentiment. In addition, Stanovsky et al. [120] illustrates MT systems’ tendency to ignore morpho-syntactic contextual cues in coreference resolution settings in favour of stereotype information.

However, it should also be noted that with growing pressure from the public and academic research pointing at biases in MT systems, there has recently been some positive development such as Google providing several possible translations in the case of words with ambiguous gender [70, 104]. Besides problems in the translations of gender-neutral (pro)nouns, another form of gender bias recorded for MT is stylistic bias. Hovy et al. [60] found that due to the demographic skew in training data, automatic translations made users sound older and “more male.” Moreover, making gender information for first-person narration salient in non-gender marking source languages improved the translation of women’s voices into gender-marking target languages [130]. In addition, there exist several challenge or benchmark datasets to assess gender bias for different MT systems, such as WinoMT [101, 120], the occupations test set [44], the Arabic Parallel Gender Corpus [55] and MuST-SHE [12].

Gender bias detection and mitigation efforts in NLP until this point have mostly employed a binary conceptualisation of gender, meaning that these works concentrated on equality in representation and quality of service for only male and female gender [38, 39]. Including other genders besides binary male and female was either not mentioned at all [71, 85], mentioned only when discussing future work [142], limitations [8, 41], or mentioned as an issue that the authors were aware of but could not be addressed, because it would complicate experiment setups [29] or the work was building on prior work with a binary conceptualisation of gender [126].

Devinney et al. [39] presented a two-round survey of conceptualisations of gender in NLP research in 2020 and 2021 and found that while awareness for and inclusion of non-binary gender models are increasing, more than half of all research surveyed still subscribed to the binary “folk” model of gender, according to which there are only two immutable categories of gender, male and female. They found most works lacking explicit definitions of the conceptualisation of gender that was applied and of how gender was implemented in experiments. In line with this, they also found that social gender and linguistic gender are often conflated. Moreover, there are few works that address intersectional aspects in connection with gender, such as race or socioeconomic status. Devinney et al. [39] recommended future publications to explicitly define gender using appropriate and respectful language, subsequently select a method in line with the chosen definition of gender, and finally base the work on feminist research.

Dev et al. [38] moreover explored harms and challenges related to the exclusion or misrepresentation of gender identities that are non-binary in the context of NLP. Because non-binary gender identities (non-binary, agender, genderqueer, etc.) are not always recognised and not well-understood in large swaths of public discourse, training datasets for language technology reflect this lack of (accurate) representation. This data deficiency leads to language models, which currently function as the basis for most state-of-the-art language technology, creating “meaningless, unstable representations” for words used to express non-binary gender [38]. For example, the neopronouns xe and ze are treated as out-of-vocabulary tokens by BERT [40]. As a result, downstream applications such as machine translation or coreference resolution systems are likely to fail at resolving neopronouns and other language for expressing non-binary gender identities. The result is either the misgendering and/or erasure of non-binary genders [38]. For future work, the authors mentioned two primary challenges: the need for more real-world data of neopronoun use and a move away from a tripartite view of social gender as male/female/gender-neutral but rather a more open conceptualisation of gender to account for its fluidity [38].

While measures for the assessment of bias have lead to an awareness of how models integrate and emphasise existing biases in the data, a definitive bias measure that works reliably, especially in the context of large-scale language models, does not yet exist.

Aribandi et al. [4] tested the SEAT [85], CrowS-Pairs [91], and StereoSet [90] on three BERT models with different random seeds. They found that, while the performance of the models remained stable, predictions of the stereotypical categories in SteroSet and CrowS-Pairs, as well as statistical significance of the SEATs, appeared to be erratic (i.e., heavily influenced by the configuration of an individual model). In addition to inconsistencies in their application, Blodgett et al. [18] moreover found that a variety of pitfalls in four bias measuring datasets themselves. They analysed StereoSet and CrowS-Pairs, as well as WinoGender and WinoBias, which all contain contrastive pairs meant to measure a model’s performance on stereotyped versus non- or anti-stereotyped examples. Within these examples, the researchers found a string of inconsistencies in the operationalisation of stereotyping, with some examples being non-meaningful, misaligned, non-relevant, or containing offensive language in place of a stereotype, among others. Blodgett et al. [18] also criticised that all datasets analysed lacked a clear conceptualisation of stereotyping, which is their main focus.

Furthermore, as mentioned earlier, it should not be assumed that when task-agnostic gender bias can be measured for an intermediate representation, such as a word embedding model or language model, that this will definitively translate to task-specific bias in the downstream application [48]. Similarly, different bias measures might not necessarily correlate for the same model [37].

Overall, these works show that while the task of being able to measure problematic behaviour in models is important, it is also equally important to carefully construct measures [18] that remain robust to different configurations of the same model, take model uncertainty into account [4], and illustrate the influences on downstream applications [37].

Gender bias in visual datasets is strongly influenced by the source of the images and the creation of the training labels. Online image hosting platforms, encyclopedias, and social networking sites are popular sources of visual data. Data-centric bias detection involves auditing and measuring gender bias in visual training datasets and is commonly performed via statistical methods, contextual representations, and empirical analysis of the datasets.

Statistical Methods. Statistical measures, such as frequency counts, are often employed to measure and analyse bias in datasets. These range from analysing demographic details, such as age, gender, and race, to using statistical techniques, such as t-distributed Stochastic Neighbour Embedding (t-SNE), to visualise the distribution of images.

Using statistical methods, Singh et al. [116] compared image retrieval results across various image search and hosting platforms such as Bing, Twitter, the New York Times, Wikipedia, and Shutterstock. They used gender-skewed occupations such as librarian, nurse, programmer, and civil engineer and found, compared with data from the U.S. Bureau of Labor Statistics on gender participation in that particular occupation, that the New York Times had the most balanced representations while Twitter had the least.

As an example of selection and labelling bias, biased data from these online platforms is then used to curate datasets for training deep learning models. Yang et al. [138] studied image representations in the ImageNet hierarchy and found gender bias in the very popular dataset. There were instances of labels having gendered and sexist slurs and pejorative keywords. Along with that, most annotations had gender bias such as the term banker having mostly male images. Yang et al. [138] used Amazon Mechanical Turk to correct some of these biases by balancing them for race and gender. Other similar debiasing techniques such as relabelling and training data augmentation have been discussed in Section 4.2.1.

Gender bias in visual datasets comes in many forms. Apart from labels, bias manifests in many implicit ways such as the depiction of gender in visual scenes. Wang et al. [132] analysed various popular vision datasets such as COCO, OpenImages, and YFCC100m. They analysed image scenes and found that outdoor scenes such as transportation, snow and ice, deserts and sky, fields and parks had more representation of men whereas indoor scenes such as shopping, dining, indoor sports, and leisure and home themes had a higher representation of women. In scenes related to objects, images under the categories sports and vehicle had more images of men, whereas those under the categories kitchen, appliance, indoor, and furniture had more images of women. Such issues can lead to framing bias in vision datasets.

Stereotypical gender representations are often over-amplified in image datasets. Kay et al. [65] studied the effect of stereotype exaggeration, systematic over- and under-representation and peoples’ perception of stereotyping of gender in image search results returned by search engines. They used occupations that have a strong gender skew for their experiments and based their hypotheses on data on occupations collected from the U.S. Bureau of Labor Statistics. Kay et al. [65] found significant stereotype exaggeration with images associated with women for traditionally female dominated occupations and vice versa. Terms such as sexy and attractive returned a very high percentage of images of women (\(\%=0.8\) and \(0.72\), respectively) and professional and trustworthy returned images of men (\(\%=0.27\) and \(0.6\), respectively). This is another example of selection bias.

Statistical measures paired with visualisations can provide useful insights into the nature and distribution of bias in data. Karkkainen and Joo [64] used t-SNE to visualise the distribution of images by race to analyse their distribution. Such visualisations are useful to understand data distribution and bias in a high-dimensional space and have the potential to be extended to gender bias.

Contextual Representations. In their analysis of the OpenImages dataset, Wang et al. [132] found that in images of people with musical instruments, men were often depicted as playing or interacting with them whereas women were more likely to be seen as audience. This meant that men were generally closer to the instruments. They found that men were more likely to be engaging with objects such as those related to sports and vehicles, and women with objects related to the kitchen, furniture, and accessories. Such representations can lead to framing and selection bias.

Empirical and Manual Analysis. Implicit gender bias is often difficult to measure using quantitative and statistical tools. Such biases are often hidden in the setting and context of the images and their associated texts and a qualitative analysis approach may be needed. Birhane et al. [15] studied the issue of harmful stereotypes in very large image datasets such as LAION-400M containing 400 million images and text extracted from the alt-text in web pages (i.e., crawled from the Internet). In the dataset they identified both harmful text and sexually explicit images and using text-based image retrieval methods found harmful images returned for terms related to women such as Maa, Aunty and Abuela. Similarly geographic biases were identified. The authors attribute this to the data creation method of crawling the Internet without filtering leading to labelling bias.

Using a similar methdology, Wang et al. [132], analysed the popular image dataset OpenImages and found that in images of people and flowers, women were more likely to be posing with flowers, whereas men would have flowers used for background decoration. They found, using a convolutional neural network (CNN) trained on OpenImages, that the model was then more likely to classify people in indoor sports, such as swimming, as women, and in outdoor sports, such as football, as men.

Benchmark Datasets. Another approach to detecting and measuring bias in CV is to create benchmark datasets to measure variance and diversity. Karkkainen and Joo [64] analysed various popular visual datasets such as IMDB-WIKI, LFW+, CelebA, UTKFace, among others, and looked into the racial and gender distribution. They found that apart from FotW and UTKFace, none of the datasets were balanced. They also created their own dataset called FairFace, which had balanced racial and gender distribution. To compare the performance of their dataset, they trained a ResNet-34 convolutional neural network on all the datasets individually and tested them on diverse sets of images such as images from geo-tagged tweets, media photographs, and protest datasets, all balanced for race and gender. They measured balanced accuracy on gender by using a variation of equalised odds to measure the difference in true and predicted gender. Karkkainen and Joo [64] found that the model trained on FairFace performed better than the models trained on almost all other datasets.

As discussed in Sections 3.1.1 and 4.2.1, benchmark or challenge datasets as a bias mitigation tool present their own challenges. In an effort to provide a more objective measure of image diversity and to assess the impact of standard image search engines on the creation of image datasets, Mandal et al. [80] used Google Image Search with queries in different languages and using different geolocation settings (via a VPN) to gather images and create a dataset. Neural networks were then used to extract visual features for comparison to judge the resulting variation between searches and the overall diversity of the images independently of any labels.

Image Embeddings Association Test. A popular methodology for bias detection in CV is to borrow and adapt techniques from other fields (such as NLP). Steed and Caliskan [121] proposed a methodology based on the Word Embedding Association Test (discussed in Section 3.1.1, Embedding Association Tests): the Image Embeddings Association Test (iEAT) to quantify implicit human biases. They hypothesised that human-like biases are present in the image embeddings used by neural networks. The iEAT measures the correlation between concepts. Using two sets of target concepts and attributes (e.g., male–career, female–family), the test measures the statistical differential association between them based on the model’s embeddings and produces a standardised measure of the probability that no bias exists.

A similar assessment of model bias was designed by Caliskan et al. [25], who used two CV models: iGPT and SimCLRv2, both pre-trained on Imagenet. The experiment for gender bias included testing the models on two tests. First, Gender-Career test that measures the relative association of the category male with career attributes like business and office, and the category female with family related attributes like children and home. Second, Gender-Science test that measures associations between male with science and engineering, and female with liberal arts and writing. They found significant gender bias in both models in the Gender-Career test and the Gender-Science test with the standardised probability values being higher for SimCLRv2 than iGPT.

The use of iEAT to measure bias in CV models is relatively recent. Sirotkin et al. [117] used the iEAT to study the effect of Self-Supervised Learning (SSL) on bias. They studied three SSL models: geometric, clustering-based, and contrastive. Using the Gender-Career and Gender-Science tests, they found that contrastive models had the highest bias.

Model Leakage. The concept of model leakage was defined by Wang et al. [134] in studying spurious correlations in vision datasets that lead to bias. For example, in the popular image dataset COCO, there are more images that contain both plates and women than there are of images that contain both plates and men. This might lead to gender bias in models that then strongly correlate plates with female gender. This ability to infer gender from unrelated predicted image labels (plate predicts female) is termed “leakage.” The “leakage” in models is measured by the percentage of examples in the dataset that “leak” information about a protected label (e.g., female) through the model’s predicted labels (e.g., plate), assessed by training a new function that aims to predict the gender from the labels.

Bias Amplification. Models might to not only reflect the bias in the dataset, as in model leakage, but increase or amplify this bias. The term was defined by Wang et al. [134] as the difference between the evaluated model leakage and the dataset leakage. Alternatively, Zhao et al. [140] measured bias amplification by comparing the effect of bias correlation learnt by a model during training. For example, in visual semantic role labelling (VSRL), labels (person, spatula, oven, etc.) are generated for a scene such as a kitchen and the resulting activity shown in the image is cooking. If the positive correlation between two terms (e.g., women and cooking) is increased by the model over the evaluation dataset, this is termed bias amplification. The total score for the model is estimated as the average magnitude for all pairs that exhibit bias. Both of these metrics aim to quantify the bias influence of a model over a reference dataset, however, there is a risk of oversimplification and of reliance on a well-annotated reference dataset to compare against.

InsideBias. An alternative view of bias measurement is to inspect the internal structures of the model such as the activation of filters in a CNN, commonly used for state of the art CV models. Serna et al. [109] proposed such an approach to measure demographic bias and evaluated it by training two CNN architectures (VGG and ResNet) on DiveFace, a diverse dataset with representations from across the world, and on biased data by increasing the representation of a particular group. To assess the impact of biased data, an Activation Ratio is calculated. Activation, a measure of the contribution of network layers in generating the feature map, is compared between networks trained differently and the resulting models are considered biased if the ratio is less than a defined threshold. Generally the final layers of the network have the highest contribution and are evaluated in this way. Serna et al. [109] found that the unbiased models had a higher activation ratio for the last layers than the biased ones supporting their claim that this approach can give a good indication of model bias.

Grad CAM. Gradient Weighted Class Activation Mapping developed by Selvaraju et al. [108] provides localised visual explanations for CV models by creating a heatmap over the input image showing the regions contributing to the classification. It is done by computing the gradients flowing into the last (pre-fully connected) layer. This concept was utilised by Reference [83] for bias analytics. They used a visual question-answering machine based on CLIP (Contrastive Language Image Pretraining) similar to Reference [113] to analyse bias in CLIP image encoder models.

Difference Metrics. In contrast to inspecting the internal structures of the model, the difference in the model’s output predictions can also be compared statistically. This is distinct from bias amplification methods (see previous section) that train a predictive function to reverse the model’s outputs and calculate the leakage or correlation. In their work on debiasing neural networks, Savani et al. [102] used a fairness metric based on the difference in outputs predicted by neural networks for different demographic groups. These include Statistical Parity Difference (SPD), Equal opportunity difference (EOD), and Average Odds Difference (AOD). True- and false-positive rates, standard metrics that measure the accuracy of a models output against the provided evaluation labels, are used to calculate the probability of positive outcomes (predictions or labels) for protected and unprotected groups. SPD measures the difference between the probability outcomes while EOD and AOD look specifically at the differences in true positive rates. Together, these metrics quantify prediction accuracy specifically focusing on protected and unprotected groups. Again, in common with other metrics, this is dependent on the quality of the reference annotations in the evaluation dataset. All these metrics are designed to work on CNNs. However, with the growing popularity of Vision Transformers (ViTs), similar metrics are required to audit bias in them. Mandal et al. [81] proposed Accuracy Difference, a metric that can measure bias in both CNNs and ViTs. The metric is a difference in accuracy between two sets of models: one trained on biased data and the other on unbiased data. The accuracy is measured on an unbiased test set. The higher the difference, the more the bias. They found ViTs to be more affected by bias due to a shallower loss landscape leading to more generalisation and global attention and a larger receptive field allowing ViTs to learn more visual features and capture longer dependencies. These factors enable ViTs to learn more biased features from images.

Similar to NLP, bias analytics in CV has focussed mainly on binary gender bias. However, unlike NLP, research on non-binary gender bias in CV is extremely limited. Luccioni et al. [78] studied the presence of intersectional gender bias in TTI models by evaluating their output using image captioning models and creating clusters based on visual features. Their tool StableBias also allowed for visual analysis of the outputs. They used prompts that included multiple identities such as occupation, ethnicity, and gender. Their tool allows for exploratory analysis of the output of TTI models but does not allow for quantitative measurement of bias, especially in the representation space. To generate a diverse dataset based on social characteristics, they used a pattern Photo portrait of [X][Y] with X and Y being characteristics related to ethnicity/gender and professional attributes. This dataset was used to evaluate three TTI models: DALL-E2 and Stable Diffusion v1.4 and v2. Then three types of analysis were done. In the first set, they performed an analysis of the text features of the generated images using two Image-to-Text models: a ViT-GPT-2 model trained on MS COCO and a VQA system based on BLIP. They analysed the text features for gender and ethnic markers for professional attributes and compared them with data sourced from labour bureaus. The analysis revealed DALL-E 2 has the highest deviation from the real-world data followed by Stable Diffusion v2 and v1.4. The second analysis involved clustering visual features extracted from the images using the same BLIP VQA used before. The results indicated that men made up most of the professional clusters. The third analysis involved creating an interactive tool to study these biases on a case-by-case basis.

As we described in Section 2.3, the training data are one of the primary entry point for gender bias into NLP models. Therefore, changing the data in a way that counters prevalent gender imbalances and stereotypes is an obvious starting point for bias reduction.

Counterfactual Data Augmentation (CDA). One of the most common methods is Counterfactual Data Augmentation (CDA). In CDA, pronouns and nouns referring to female (male) gender are swapped for those referring to male (female) gender. For English text, Lu et al. [77] appended sentences edited in such a way to the original training data, thereby augmenting the corpus. Maudslay et al. [84], however, substituted the original sentence for the augmented one, which they called Counterfactual Data Substitution (CDS). They additionally developed a method of also swapping first names in such a way that name frequency (James vs. Bart) and gender-specificity (Anna vs. Taylor) were preserved. Bartl et al. [8] applied CDS to fine-tuning data for English BERT and demonstrated bias reduction using the log probability bias score by Kurita et al. [71]. Dinan et al. [43] and Webster et al. [137] moreover demonstrated the usefulness of CDA for bias mitigation on dialogue generation tasks. While the previous works have focused on English, Zmigrod et al. [145] showed that CDA is also useful for reducing gender stereotyping in gender-marking languages (Hebrew, Spanish, French, Italian) and moreover provided a method for adjusting the gender of dependants of a swapped instance according to the “new” gender.

Gender Neutralisation. Instead of swapping gendered words, another strand of research targeting the training data concerns the creation of gender-neutral text. Vanmassenhove et al. [129] and Sun et al. [125] both developed applications designed to turn gender-specific into gender-neutral sentences. For instance, the sentence “Every stuntman accepts a considerable risk of injury in his job.” would be turned into “Every stunt performer accepts a considerable risk of injury in their job.” The researchers mentioned machine translation as one possible area for the application of post-hoc gender-neutralisation. Vanmassenhove et al. [129] additionally mentioned that gender-neutral text created by their system could also be used to mitigate bias in training data, but left this to future research.

Model-centric debiasing techniques have traditionally focused more closely on mitigating semantic bias, meaning bias in intermediate textual representations, than bias in the models themselves.

Word Embeddings. As one of the earlier works on illustrating gender bias in word embeddings, Bolukbasi et al. [19] presented a method called direct debiasing, with the rationale of removing associations with gender from the embeddings. They obtained a gender subspace through combining the vectors for a variety of words containing masculine and feminine gender, such as pronouns, then projected the word vectors onto this subspace, and subsequently removed the projections from the original vectors.

Zhao et al. [142] critiqued Bolukbasi et al.’s [19] method for completely removing gender information from word embeddings, which might not always be desirable. To overcome this flaw, they presented a method for learning gender-neutral GloVe word embeddings [94]. Their approach concentrated gender information in specific dimensions of the vectors, which could then be removed to reduce biased gender associations but preserve factual gender information. Using Bolukbasi et al.’s [19] gender direction, they illustrated a reduction in gender stereotyping and moreover showed a reduction in bias on a coreference resolution task [142].

In a seminal work, Gonen and Goldberg [49] then assessed the effectiveness of the two previously described debiasing techniques by Bolukbasi et al. [19] and Zhao et al. [141]. They stated that the way bias removal is conceptualised in these works, that bias is removed if definitionally gender-neutral words all have an equal distance to all pairs of explicitly gendered words, ignores more implicit associations relating to gender stereotypes. In their experiments, Gonen and Goldberg [49] used clustering and gender prediction to show that stereotypical gender information can still be easily recovered from de-biased embeddings.

A method using projections to remove bias, Iterative Nullspace Projection, was proposed by Ravfogel et al. [99]. They trained a linear classifier to learn the direction corresponding to attributes of a protected group. Then, to debias, they projected the sentence representations into the nullspace of the linear classifier’s weight matrix, thereby removing information about the protected group.

Large Language Models. Since the emergence of transfer learning from pre-trained, transformer-based LLMs and their widespread adoption within the field of NLP, the most recent efforts at debiasing NLP models have focused on language models.

One approach taken by Liang et al. [74] was the adaption of Bolukbasi et al.’s [19] direct debiasing technique for LLMs, which is called SentDebias. Liang et al. [74] contextualised a set of identity terms and terms related to a protected group, gender in this case, in randomly extracted sentences, estimated a bias subspace from these sentence representations, and subtracted the projection onto the subspace from the LLM’s representations.

Webster et al. [137] targeted model bias that is created through the exploitation of spurious correlations with gendered identity terms. To reduce those correlations, they increased the dropout parameter during additional pre-training for BERT [40] and ALBERT [72]. Dropout regularisation is normally used to avoid overfitting, but Webster et al. [137] showed that the effect of reducing superfluous correlations also reduces correlations that express stereotyping in masked LLMs, while keeping performance consistent.

Opposed to the previous methods, which essentially change the model’s internal representations, Schick et al. [105] proposed a post-hoc method called SelfDebias. They base their method on the observation that LLMs are able to detect when their own output contains toxic or biased text, which they call self-diagnosis. Based on this observation, they then first prompted the model to create a text containing a form of bias, and subsequently de-biased by scaling down the probabilities for the generated biased text for a secondary generation of text.

Guo et al. [54] presented their approach called AutoDebias. It is similar to Schick et al.’s [105] SelfDebias, however, instead of crafting prompts to elicit biased text, Guo et al. [54] automatically found prompts that can be used for de-biasing a masked language model (MLM), thereby reducing the reliance on external corpora. They choose prompts “such that the cloze-style completions have the highest disagreement in generating stereotype words (e.g., manager/receptionist) with respect to demographic words (e.g., man/woman)” [54]. These prompts were then used to fine-tune the MLM in such a way that the disagreement between the two generations for binary gender words are minimised. Guo et al. [54] showed that this kind of de-biasing does not hurt model performance on the GLUE benchmark.

Limisiewicz and Mareček [75] aimed to preserve factual gender information while removing gender bias from the top layer of pre-trained, transformer-based language models. They used an orthogonal probe to distinguish between gender associations related to factual gender versus gender bias, and then filtered out the bias subspace from the embedding space. They showed that, while not all of the stereotypical bias is removed, their method succeeded in mitigating bias while preserving language modelling ability [75].

Garimella et al. [47] presented another approach to bias mitigation by introducing two additional loss functions during additional pre-training, an equalising loss and a declustering loss, aimed to “equalize the associations of words with different groups of a given demographic” and “decluster the various clusters of words that may be indicative of certain kind of implicit bias with respect to the demographic” [47]. They evaluated this method on a BERT model using SEAT [85] as well as human evaluations and found sentence completions to be less biased. In addition to bias reduction during pre-training, the researchers also presented a bias mitigation objective during decoding for a specific language generation task, text summarisation in this case.

Meade et al. [87] compared several of the previously presented gender bias mitigation techniques for LMs: Dropout regularisation [137], SentenceDebias [74], SelfDebias [105], Iterative Nullspace Projection [99], and CDA [77, 145]. Measuring bias with the SEAT, StereoSet, and CrowS-Pairs, they found SelfDebias [105] to be the most effective technique, that also consistently preserved language modelling ability.

Data-centric bias mitigation techniques involve modifying the training data to either have unbiased training datasets or use of specific datasets to de-bias existing models.

Relabelling. The most straightforward method to reduce data-centric bias is to relabel or refine the existing annotations and classifications. This can potentially mitigate framing, labelling, and selection bias. Relabelling can be expensive, time-consuming and require significant domain expertise but allows the utilisation of existing large image collections. OpenImages contains about nine million images that contain five person-level annotations: person, man, woman, boy, and girl. Schumann et al. [106] studied the gender bias in these annotations and proposed a new framework called More Inclusive Annotations for People (MIAP). For example, they found that in images containing both men and women in settings such as weddings, the bounding box focused on women whereas it was reversed in case of images depicting military personnel. They introduced MIAP to replace the five person-related keywords with three terms: predominantly feminine, predominantly masculine and unknown in an effort to mitigate these effects.

Training Data Augmentation. One of the more fundamental approaches to bias mitigation involves modifying the training data with respect to model behaviour. Zietlow et al. [144] used an Adaptive Sampling method to improve fairness in vision classifiers. They started with two sets of training data: an original set and an extended set. They trained a classifier on the original set and determined the worst performing group using a hold-out dataset. Then they added the group to the extended dataset and measured the resulting changes in the classifier’s performance using a sampling approach (g-SMOTE) that promotes oversampling of minority classes from the data. The results of their experiments showed a considerable improvement of the model when retrained with the augmented data with increasing representation of the worst performing group. This method also outperformed other fairness techniques such as weighing and fairmix [144], and help mitigate selection bias.

Benchmark Datasets. Rather than trying to improve an existing labelled dataset, benchmark datasets, such as the FairFace dataset [64], are specifically created to serve as a standard against which training data can be checked. They aim to provide a reference for gender and racial diversity. Similar datasets such as UTKFace [139] and the Pilot Parliament Benchmark [23] have also been proposed. However, these benchmark datasets risk incorporating their own selection bias, most prominently a Western-centric bias on how race and gender are conceptualised, and the authors proposed various methods to avoid compounding such bias. A dataset creation process where a variety of terms in different languages and different geolocations for Google Image Search was used in Mandal et al. [80] to assess the impact of the dataset creation choices on bias. Such datasets can help mitigate selection bias.

Model-centric bias mitigation techniques generally involve targeting the internal representation learnt by the model in its embedding space. These include modifying the training objective function to focus on debiasing and adversarial debiasing. An overview of model-centric bias mitigation techniques in CV is provided in Table 6.

Learning Representations. Sampling techniques, such as those used in the previous approach, come with their own issues including potential for oversampling and overfitting. Wang et al. [135] proposed two techniques to overcome these issues: Domain Discriminative Training and Domain Independent Training, which can help mitigate sampling bias. Domain discriminative training (DDT) works on the opposite principle of the “fairness through blindness” concept. In DDT, information is first encoded and then mitigated. The model first learns correlations between the target class and the domain that leads to bias (such as man–programming and woman–cooking). Then the model is trained to minimise these correlations to reduce bias. In Domain Independent Training (DIT), the model learns these correlations but is trained to ignore these class boundaries. The authors tested both the methods on the Celeb-A dataset. Their aim was to remove gender bias by using a weighted mean average precision (mAP) metric to simulate equally distributed samples between the genders. They found that the DDT model performed worse than the base model (73.8% vs. 74.7% mAP), while the DIT performed better with 76.3% mAP.

Model Fine-tuning. Large pre-trained models work well on general CV problems. They are however difficult and expensive to train. Therefore, any significant retraining is expensive and time and resource consuming. This has led to the development of fine-tuning algorithms where models are retrained to achieve specific goals including for debiasing. One such method is Adversarial Debiasing proposed by Wang et al. [134] to reduce model leakage (see Section 3.2.2 model leakage) by discouraging the model from building representations from protected attributes such as gender. They construct a critic model that tries to predict protected attributes from an intermediate representation for an image from a competing predictor model. In its simplest form, the predictor tries to improve classification performance at the expense of the critic (meaning that the critic’s ability to predict protected attributes decreases) to result in a more balanced and less biased system. The authors also experimented with optimisation of the adversarial loss on the input feature space by using an encoder-decoder model and auto-encoding input image.

Wang et al. [134] used three types of adversaries to remove leakage at different stages in a ResNet-50 classification model. The first targets the image directly, trying to remove gender information by using a U-Net as the encoder-decoder network to predict a mask. The second removes gender information from an intermediate representation of ResNet-50 (the fourth convolutional block) using an adversary having three convolutional layers and four linear layers. The third method removes gender information from the last convolution layer of ResNet-50 using a linear adversary taking a vectorised form of the output feature map as input and a four-layer MLP as classifier. These approaches try varying methods of crafting suitable adversaries (the critic and the predictor) based on the image or specifically targeting a layer of the model. Various models are used as the base model, trained on original data, and models trained on different augmented data such as with Gaussian blur, face blackout and blur. From their experiments, Wang et al. [134] found that the three adversarial models resulted in less bias amplification than the baselines with the second approach (targeting an intermediate representation layer) performing the best.

A second approach to model fine-tuning are Intra-processing Methods. As demonstrated by the success of the adversarial method targeting an intermediate layer, it is possible to debias pre-trained models by focusing on CNN layers. Savani et al. [102] proposed intra-processing algorithms for debiasing vision models trained on large generic datasets, as a complement to in-processing methods. They propose three intra-processing algorithms: random perturbation, layer-wise optimisation, and adversarial fine-tuning, which we discussed above as an example of adversarial debiasing.

The intra-processing algorithm takes the validation dataset and a trained model with a set of weights and outputs a fine-tuned weights set that optimises the desired outcome. The authors proposed the following intra-processing debiasing algorithms that optimise metrics similar to the difference metrics explained in Section 3.2.2, which are based on true and false positive rates. Random perturbation is an iterative algorithm in which every weight in the network is replaced by a Gaussian random variable (mean of 1 and standard deviation of 0.1) in every iteration. This aims to disrupt the training and force the layers to avoid over-generalisation that can lead to bias. Layer-wise optimisation debiases the model by debiasing individual layers using a more reliable means of finding an optimum network point, and that can only operate on a feed-forward neural network. In their experiments, Savani et al. [102] used Gradient Boosted Regression Trees as the optimiser. The authors found significant bias reduction using a ResNet model tested on the CelebA dataset with the Layer-wise optimised model outperforming the random perturbations, again showing the advantages of a more targeted approach.

A third technique for model fine tuning is Reducing Bias Amplification (RBA). Deep neural networks learn representations in the data by creating correlations between the features in the input. This can lead to the network amplifying certain correlations that may then amplify any bias present in the training data. Zhao et al. [140] proposed RBA to reduce bias arising out of spurious correlations in visual datasets. Details in images can often contain features introducing bias in models trained on them (as explained in Section 3.2.2: Bias Amplification). The authors here aimed to mitigate such biases by injecting constraints to make sure that the model follows the distribution present in the training data. The proposed algorithm is a meta-algorithm based on Lagrangian relaxation consisting of three main parts. The first part involves structured output predictions, where a scoring function is created based on the model and decomposed to extract the part concerned with semantic labelling. In the second part, corpus level constraints are introduced to ensure that the output labels follow a desired distribution. For example, the gender ratio for each activity can be constrained. The third part involves solving this constrained problem, expressed as an integer linear program – a set of linear constraints over integer variables, by using a solver (the authors used Gurobi Optimisation in their experiments).

This algorithm was evaluated on two tasks: visual semantic role labelling (vSRL), and multi-label classification (MLC). They focused on gender-specific terms (man and woman) and the agent in vSRL and text association with images in MLC. For vSRL, they used the imSitu dataset containing about 125,000 images with activity classes drawn from FrameNet and noun categories drawn from WordNet. Non-human activities were filtered out. They build a Conditional Random Field (CRF) model for testing. For MLC, they used MS-COCO, an object detection benchmark containing 80 different object types and no gender related captioning. They used a CRF based on ResNet-50 as the model. Both the datasets are biased toward men with 64.6% and 86.6% for imSitu and MS-COCO, respectively. The results showed that the debiased models had bias reduction as compared to the baseline models.