Watching the Watchers: A Comparative Fairness Audit of Cloud-based Content Moderation Services

Abstract

Online platforms face the challenge of moderating an ever-increasing volume of content, including harmful hate speech. In the absence of clear legal definitions and a lack of transparency regarding the role of algorithms in shaping decisions on content moderation, there is a critical need for external accountability. Our study contributes to filling this gap by systematically evaluating four leading cloud-based content moderation services through a third-party audit, highlighting issues such as biases against minorities and vulnerable groups that may arise through over-reliance on these services. Using a black-box audit approach and four benchmark data sets, we measure performance in explicit and implicit hate speech detection as well as counterfactual fairness through perturbation sensitivity analysis and present disparities in performance for certain target identity groups and data sets. Our analysis reveals that all services had difficulties detecting implicit hate speech, which relies on more subtle and codified messages. Moreover, our results point to the need to remove group-specific bias. It seems that biases towards some groups, such as Women, have been mostly rectified, while biases towards other groups, such as LGBTQ+ and PoC remain.

1 Introduction

The digital age has brought a significant increase in online content. Worryingly, this also includes pernicious, unwanted content, such as hate speech [1]. Online platforms responded by adopting extensive content moderation regimes [2]. Absent a legal taxonomy of what constitutes hate speech, private companies are given substantial autonomy in their moderation practices, effectively making them the judges of public speech [3, 4].

Typically, large online platforms rely on so-called community guidelines against which speech is assessed. The assessment is done by human moderators, who are assisted by algorithms [5]. The largest technology firms, such as Google, Microsoft, and Amazon, additionally offer content moderation as a service via cloud-based API access. While most organisations do not report the extent to which algorithms shape content moderation, the sheer amount of online speech makes reliance on algorithmic moderation inevitable [6].

To date, there exists no systematic evaluation of cloud-based content moderation services. The absence of public scrutiny is alarming, as open-source content moderation algorithms have continuously displayed biases against minorities and vulnerable groups [7, 8, 9, 10, 11, 12]. This paper’s contribution is twofold. Firstly, it offers the first comprehensive fairness assessment of four major cloud-based content moderation algorithms. Not only are these algorithms likely in use at the companies themselves, but they are also deployed by a vast number of smaller organisations through the SaaS model. Secondly, our auditing strategy may inform future bias audits of (cloud-based) content moderation algorithms. Importantly, our proposed approach solely assumes limited black-box access [13] and offers guidance on reinforced sampling strategies to achieve maximal scrutiny with limited resources, noting the realities of unsolicited audits from civil society organisations and academia [14, 15, 16].

2 Data and Method

We gained researcher access to the Google Moderate Text API, Amazon Comprehend, Microsoft Azure Content Moderation, and the Open AI Content Moderation API. These services generate a hate speech score per text sequence, often split across several sub-categories, as well as a binary flag. Our study uses the MegaSpeech, Jigsaw, HateXplain, and ToxiGen datasets [17, 18, 19, 11]. The selected datasets capture various forms of hate speech, with ToxiGen containing implicit and adversarial hate speech constructed around indirect messages [20], while MegaSpeech and ToxiGen use generative AI to diversify speech corpora [17, 11]. Jigsaw and HateXplain contain human-written examples labeled by annotators, with MegaSpeech containing more hate speech corpora but no target group labels. MegaSpeech, HateXplain, and ToxiGen provide shorter text sequences, with on average 17.7, 23.3, and 18.1 words respectively, while Jigsaw is made up by longer sequences, 48.3 words on average.

We evaluate all cloud-based moderation algorithms across all datasets on a set of threshold-variant and threshold-invariant performance metrics [21, 22] at an aggregate level and also specifically for vulnerable groups. We ensure consistency across datasets by mapping these onto seven vulnerable groups (Women, LGBTQ+, PoC, Muslim, Asian, Jewish, Latinx). Since MegaSpeech comes without labels, we train a Bi-LSTM model with the collected data set by Yoder et al. [23] (preliminary evaluation accuracy 78 %) for target identity classification.

Dataset	Moderation Service	ROC AUC	F1	FPR	FNR	Dataset	Moderation Service	ROC AUC	F1	FPR	FNR
ToxiGen	Amazon	70.4%	68.9%	7.2%	52.0%	MegaSpeech	Amazon	72.8 %	72.0 %	10.4 %	43.9 %
	Google	62.7%	62.7%	39.1%	35.5%		Google	73.3 %	72.3 %	41.3 %	12.0 %
	OpenAI	70.3%	68.1%	33.2%	56.0%		OpenAI	77.1 %	76.7 %	8.4 %	37.3 %
	Microsoft	59.8%	57.4%	16.4%	64.0%		Microsoft	70.6 %	70.1 %	16.9 %	41.9 %
Jigsaw	Amazon	92.2%	92.2%	7.5%	8.1%	HateXplain	Amazon	66.8%	66.25%	46.3 %	20 %
	Google	69.9%	67.2%	58.4%	1.8%		Google	52.2 %	58.9 %	78.2 %	4 %
	OpenAI	78.6%	78.6%	17.1%	25.6%		OpenAI	72.9 %	76.7 %	45.4 %	8.86 %
	Microsoft	75.8%	75.7%	20.4%	28.1%		Microsoft	63.1 %	60.2 %	63.6 %	10.3 %

Table 1: Performance metrics by moderation service and dataset. Blue shading signals the best performance, while red shading indicates the worst performance. ToxiGen includes 7,800 observations and HateXplain 14,000, while Jigsaw and MegaSpeech each contain 50,000. All datasets are balanced on toxic and non-toxic phrases.

At the group-level, we compute the pinned ROC AUC, a metric proposed by Dixon et al. [24], designed to provide a more robust measure for scale-invariant performance comparison across sub-groups. While this approach comes with its pitfalls, as the authors themselves note in a subsequent paper, it is the best scale-invariant metric to date when presented with group-level variation in biases [22].

Perturbation Sensitivity Analysis (PSA) offers an additional, arguably more robust evaluation of group-level biases by using counterfactual fairness evaluation[25]. We follow prior research in defining an anchor group against which other groups are compared [25]. Using the dominant majority group as baseline, Counterfactual Token Fairness (CFT) scores are computed as the difference in toxicity between the baseline and the corresponding minority group. PSA makes two assumptions: (1) counterfactual pairs should convey the same or neutral meaning, avoiding any implicit biases or derogatory connotations. While constructing toxic counterfactuals is theoretically possible, it is methodologically demanding and exceeds the scope of this project. Instead, we construct 34 neutral counterfactual pairs. Importantly, each minority group is represented by multiple tokens, reflecting its different semantic representations. For instance, the minority group female also manifests as woman and women. Furthermore, (2) there should be no unique interactions between a particular minority token and the context of the sentence that would skew the analysis. This is challenging in real-world applications, as certain combinations might evoke stereotypes or specific cultural connotations. Thus, the project uses data consisting largely of short and explicit statements. Furthermore, CFT scores are calculated separately for toxic and non-toxic statements, with the latter generally supporting the assumption of counterfactual symmetry more consistently.

PSA experiments are conducted using two distinct data sets. First, the synthetic Identity Phrase Templates from Dixon et al. [24] are used. The set contains 77,000 synthetic examples of which 50% are toxic. These avoid stereotypes and complex sentence structures by design, which ensures that the symmetric counterfactual assumption is met. Mapping the dataset, which contains a broader set of identities, to the 34 minority token relevant to this study, results in 25,738 sentence pairs. Second, by applying the same logic, 9,190 sentence pairs are derived from the MegaSpeech dataset.

3 Preliminary Results

Table 1 shows aggregated performance results for chosen benchmark data sets. Our results indicate notable disparities between moderation APIs. OpenAI’s content moderation algorithm performs best for Megaspeech and Amazon Text Moderation on Jigsaw and ToxiGen, generalising well across data sets. On Jigsaw, Amazon Comprehend performs best. However, its near-optimal performance (92.2 % ROC AUC) suggests that the Jigsaw data was likely included in training process of the Amazon Comprehend API . Overall, Google’s API shows the worst performance across data sets. Its poor performance seems driven by a comparably high FPR, which suggests that the algorithm tends to overmoderate. In contrast, Microsoft Azure Content Moderation is associated with a high FNR, suggesting it often misses hate speech.

Furthermore, all services struggle to detect implicit hate speech, reflected in their high False Positive Rates on ToxiGen. To this end, commercial moderation services do not fare much better than their open-source counterparts [11]. One likely cause is the limited availability of implicit hate speech datasets for training purposes.

The comparative fairness evaluation of the identity group is presented via group-level pinned ROC AUC scores in Figure 3. We find that all services tend to overmoderate speech concerning groups PoC and LGBTQ+. This is somewhat surprising as extensive prior research uncovered biases in open-source content moderation algorithms in relation to these groups [26]. Commonly, such overmoderation occurs as toxic speech concerning these groups is overrepresented in the training data, and subsequently learned by the model. Most services fail to reliably detect hate speech aimed at groups Disability, Asian, and Latinx. Lastly, the tendency of Google Text Moderation to overmoderate is puzzling but also alarming. While we cannot entirely rule out an error on our end, this observation is robust to different configurations of API sub-categories.

figureOn the left, Pinned ROC AUC is presented by moderation service, dataset and minority group. ToxiGen includes 4,268 observations, HateXplain includes 1,748, Jigsaw consists of 19,228 observations and MegaSpeech is comprised of 33,886. On the right, CFT scores are visualized. They are computed through PSA on synthetic data from the Identity Phrase Templates in Dixon et al. [24] and non-synthetic data from MegaSpeech, averaged per group and service, reported separately for non-toxic and toxic examples. Besides a point estimate, the figure also includes a 95% confidence interval assuming a student-t distribution.

Figure 3 (right) displays the PSA results. We find (1) differences in toxicity scores by and large are more pronounced on non-toxic than toxic data and (2) greater variation in the mean CFT scores in non-synthetic than in synthetic data. Overall, the results suggest that most minorities are associated with higher levels of toxicity than dominant majorities, although these effects appear relatively small, and vary across groups and services. Group LGBTQ+ seems associated with the strongest negative bias, occurring for all samples and services. We observe limited negative bias against groups Latinx and Asian.

Summarizing, we uncovered both aggregate-level performance issues and group-level biases in major commercial cloud-based content moderation services. Importantly, while some shortcomings extend to all services, such as difficulty in detecting implicit hate speech or biases against group LGBTQ+, others are confined to a particular service.