Supporting Human Raters with the Detection
of Harmful Content using Large Language Models

Content or object refers to any user-generated data, such as comments, reviews, posts, and more. For the purposes of this work, we focus on text-based content, though emergent multi-modal LLMs would also allow for the classification of images, videos, audio, and other content.

A policy is any set of natural language rules that a platform uses to determine whether or not content should be prohibited for safety reasons. Examples include policies against scams, hate speech, and misinformation.

We refer to the outcome of a policy determination as a verdict. A verdict may deem the content violative, thus triggering an escalation such as the removal of the content from the platform, or non-violative resulting in no further action. The choice of escalation is context dependent and outside the scope of this work.

We assume the existence of a human rater: a human expert responsible for making policy decisions. In practice, their expertise—and thus accuracy—may vary due to ambiguity, necessary local context, a spectrum of familiarity with the policy, rapidly changing policies, or complex cases with competing policy interest (e.g., free speech concerns vs. potential for harm).

We refer to any LLM agent that assists with policy determinations as an LLM rater. As we discuss shortly, the assistance this LLM rater provides can vary in purpose: it can automate decision making, identify errors produced by human raters, or augment the decision making process of human raters.

We assume the existence of a rater queue that includes content—or objects—that should be evaluated against one or more policies.

Human rating is a finite and expensive resource. Assuming that the cost of querying an LLM is less than that of querying a human rater, one can scale the volume of reviews beyond what was previously feasible by using LLMs to prioritize what content requires the expertise of human raters, meanwhile automating the bulk of rote decisions. Any scaling in practice is context dependent. As a rough feasibility estimate, Google Cloud charges $0.0005 per thousand characters of text input and output to PaLM 2;¹¹1https://cloud.google.com/vertex-ai/pricing\#generative\_ai\_models Microsoft Azure charges $0.001 per thousand input tokens and $0.002 per thousand output tokens for GPT-3.5.²²2https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/ While there is limited public information on human rater salaries, data annotation services such as SageMaker charge $0.08 per object³³3https://aws.amazon.com/sagemaker/data-labeling/pricing/—on the order of 100x compared to LLMs.

Human rating incurs a decision delay $t_{m}$ on the order of seconds to minutes (not including time spent in a queue). Given the latency of querying an LLM as $t_{m^{\prime}}$ for the same object, we can estimate the reduction in decision latency as $t_{m}-t_{m^{\prime}}$.

Human rating, despite relying on experts, is still subject to errors. If an LLM rater can outperform a human rater at identifying violations, this can lead to an overall reduction in false negatives (e.g., incorrectly missed violations) and potentially more consistent enforcement.

As above, if an LLM rater can outperform a human rater at identifying non-violations, this can lead to an overall reduction in false positives (e.g., incorrectly escalated violations) and potentially more consistent enforcement.

For rater queues that are heavily skewed towards non-violative content (e.g., noisy user reports or rare incidents), an LLM rater can operate as a pre-filter before sending content to a human rater. Here, the LLM rater needs to achieve a minimum threshold $T_{r}$ of recall⁴⁴4We use the standard machine learning definition of recall—or sensitivity—which is correctly identified violative samples divided by the total number of violative samples: $TP/P$. to ensure that a majority of violative content is forwarded to a human rater, while also achieving a high degree of specificity⁵⁵5We use the standard machine learning definition of specificity, which is the total number of correctly identified non-violative samples divided by the total number of non-violative samples: $TN/N$. to ultimately dequeue non-violative content. Pre-filtering serves to free up raters to review more likely violative content while increasing the total volume of content reviewed (M1). It may also potentially expedite decision making if the rater queue is non-saturated after filtering (M2).

For rater queues that may contain high-severity or emerging threats that requires immediate escalation, an LLM rater can operate as a fast-path escalation, entirely bypassing any human rater (or putting objects at the head of a queue). Doing so requires that the LLM rater achieves a minimum threshold $T_{p}$ of precision⁶⁶6We use the standard machine learning definition of precision, which is correctly identified violative content divided by all content flagged as violative: $TP/(TP+FP)$. and a high degree of recall to capture as much high-severity content as possible. Such a system might be deployed temporarily in response to an incident despite low precision (with a higher risk of appeals), or long-term if the precision is sufficiently high. This escalation architecture serves to expedite decision making (M2) and potentially avoid false negatives that a human rater may make (M3) if the LLM rater can outperform a human rater in terms of precision and sensitivity. For sufficiently large volumes of violations, it may also allow additional scaling (M1).

For some policies, it may be possible for an LLM rater to achieve equivalent—or better—performance to human raters, thus enabling the full automation of a rater queue. This represents a more difficult challenge as the LLM must achieve both a minimum threshold for precision $T_{p}$ and recall $T_{r}$, with both thresholds ideally exceeding the performance of human raters. As a safeguard, any appeals of LLM rater verdicts by users should not be sent to the same LLM rater. This operation mode has the potential to improve all performance dimensions of the rater queue.

Rather than automating (some) decisions in a rater queue, an LLM rater can serve as a quality control system. Mechanistically, an LLM rater and human rater both rate the same content. If they disagree on the verdict—and the LLM rater has sufficient confidence—the same content can be sent to one or more additional human raters. The final verdict then consists of a majority vote across human raters. This quality check can bias towards identifying errant non-violative verdicts (M3) or alternatively errant violative verdicts (M4). Configuration of the quality check requires balancing the total number of additional human ratings an LLM rater can trigger versus the lift in accuracy.

An LLM rater can also help human raters understand the context of potential violative content. During rating, a human rater might engage with an LLM rater to receive an explanation of why content is violative or not—such as highlighting specific passages of content that may violate a policy. This augmentation can help expedite rating (M2) while also drawing attention to critical context that might otherwise be missed, improving overall accuracy (M3, M4). However, there is also the risk of biasing human raters to agree with the LLM rater.

Based on early experimentation, we opted to seed all our prompt variants with a policy description derived from the harmful content policies shared publicly by Google [13]. We reformatted all of these policies into an enumerated list that is easily interpretable by an LLM. We share an example in Figure 3 and provide the specification for all four policies we consider in the Appendix.

One variable we consider is whether an LLM requires examples of comments and verdicts to render a decision. In our zero-shot variant, we provide no examples but prepend a “dummy” policy and question-answer pair to prime the LLM on our expected output format (see Appendix). For our few-shot variants, we provide a list of sample comments along with their human rated verdict, an example of which we show in Figure 4. We opt for five examples: three comments that violate the policy under consideration and two that do not violate any policy.

We explored three options for selecting these examples: (1) first, we used static, hand-picked examples. We chose these examples to be representative of the overall problem space and the policy under question; e.g., for hate speech we chose violative examples that targeted different protected groups. After observing the success of this strategy (as discussed below in Section 5.2), we next experimented with (2) dynamically identifying semantically “close” comments in terms of their embedding distance; and (3) purposefully injecting errors into the dynamically chosen samples to understand the sensitivity of prompts to noise.

Another variable we consider is whether providing context on the most salient terms from violative comments assists an LLM with rendering an accurate verdict—a form of chain-of-thought [54]. As part of our hand-picked few-shot examples, we include (manually chosen) keyword context as shown in Figure 4.

Finally, we consider how different models affect performance. We query all of our prompt variants using both the PaLM 2 text-bison and text-unicorn models (with the latter being larger than the former), available via Google’s Vertex AI API [12], each with a context window of 8192 tokens. For all experiments we use a fixed temperature of zero, which means the LLM always picks the most probable next token, and generate one token for the LLM response. This determinism ensures that an LLM produces the same output across subsequent executions, which is critical for operating a classifier in practice and for evaluating errors that might otherwise be transient. We parse the LLM output, treating “Yes” (i.e., 1) as a violative verdict and “No” (i.e., 0) as non-violative verdict. While we use only this binary output in this section, we discuss in Section 7 the potential benefits of having the LLM output more information, such as a list of keywords and the policy entry that most contributed to its decision.

For privacy reasons, we were unable to evaluate using GPT-4 or Claude-2. Likewise, Gemini Pro and Ultra had not been released at the time of our experimentation. We emphasize that our goal is to assess the general feasibility of content moderation via LLMs, rather than benchmarking the relative merit of different LLM providers. Additionally, individual LLM idiosyncrasies stemming from instruction tuning and reinforcement learning would likely require adjusting prompts per LLM architecture. Furthermore, it is cost prohibitive to query many models with all the prompt variants we evaluate in this study; as it is, with only two models we already perform half a million queries.

To compare the performance of models, we calculate a McNemar’s test statistic comparing the best performing prompt for both text-unicorn and text-bison as evaluated on the same test set. Overall, text-unicorn outperformed text-bison, with a relative lift of 0.2%–2.4% between the best prompts ($p<0.001$, except for hate speech where $p=0.177$). This modest performance gap indicates that platforms can opt for smaller and less costly models for some policies (especially keeping in mind that text-bison is an order of magnitude less expensive than text-unicorn, costing $0.00025 per thousand input characters as compared to $0.0025), but that the scaling laws of LLMs [23] still favor larger models for accurate policy interpretation.

To compare the performance of prompt variants, we calculate a McNemar’s test statistic with the zero-shot prompt as the baseline. We find that our few-shot prompt yielded statistically equivalent or better performance overall for both text-bison and text-unicorn, with the exception of harassment where hand-picked examples sent to text-unicorn degraded accuracy by 1% ($p<0.001$). On the other hand, examples carry an additional cost in terms of increasing the length of the prompt (in the case of violent extremism, which has the shortest policy specification, by as much as 61%) and thus the cost to evaluate the comment.

Keyword context appears unnecessary for text-unicorn, but does yield a 6.2% gain for violent extremism, and a 1.1% gain for harassment for text-bison $(p<0.001)$. At the same time, keyword context lifted the recall of all policies with text-unicorn—at the cost of precision. This suggests that keyword context can be omitted for LLM raters designed to automate decision where accuracy is more important than recall. Keywords may still play a role for smaller models.

A final dimension we explore is the effect of longer, possibly extraneous text on classification accuracy. Of the comments in our dataset, 50% have 11 or fewer words (and 59 characters) and 90% have 51 or fewer words (and 282 characters). We present a breakdown of accuracy for an LLM rater configured to use text-unicorn with our few-shot prompt across policies, stratified by character length in Figure 5. We find that our LLM rater declines in accuracy for longer comments. This may be a consequence of LLMs not giving enough attention to short violative passages in comments that are otherwise innocuous [30]. A potential solution would be to segment long-form text and query individual passages, but this is likely prohibitively expensive if the prompt instructions remain longer than the text under evaluation.

A pre-filtering LLM rater aims to remove clearly non-violative content from a rating queue, while falling back on human rater expertise for borderline or egregious content. An LLM rater could remove 54.5–98.2% of comments from our simulated rater queue due to being non-violative, while ensuring 99% of violative content is sent to a human rater (Table VII). If constraints are relaxed and the LLM rater need only achieve a recall of 95%, then an LLM rater could remove 85.2–99.3% of non-violative comments from our simulated queues.

A rapid escalation LLM rater aims to escalate clearly violative content without requiring verification by a human rater, while falling back to human raters on all other content. An LLM rater could automatically escalate 48.8–99.7% of violative content with a precision of 95% (Table VIII). A precision threshold greater than 99% is not feasible for hate speech and harassment, likely due to our ground truth incorrectly labeling some hate speech and harassment as non-violative. For the remaining policies, an LLM rater could automatically escalate 41.5–97.4% of violative content.

An autonomous LLM rater aims to handle ratings without any fallback to human raters. In practice, such an LLM rater achieves only 90.1–98.2% of the accuracy of human raters (Table IX). As such, the aforementioned collaborative design patterns may be more suitable given the current capabilities of LLMs.

Our validation LLM rater aims to improve overall decision making accuracy by identifying likely human errors. For our simulated queues, we reflect on the 80,000 rating decisions an autonomous LLM rater would have made and compare them to our ground truth labels from human raters. Our LLM rater disagreed with human raters on 5,896 comments, or 7.4% of the time. We randomly sampled 400 points of disagreement, stratifying this selection to include 100 comments for each simulated policy rating queue. We then manually reviewed each comment using the same methodology as in Section 4.2 to determine whether researchers agreed with the policy expert or the LLM rater. We treat these as tie-breaking ratings.¹⁰¹⁰10As with our earlier validation, we caution researchers are not trained policy experts, so concurrence with an LLM may be inaccurate.

We present our results in Table X. For hate speech, our tie-breaking rating agreed with LLMs 49% of the time, versus the ground truth 43% of the time. As an illustrative example, our LLM rater flagged “nice [thumbs up emoji] 88”, a Nazi reference, which was potentially missed by the human rater. However, our LLM rater missed “Cull these knickers”, a likely slur. For election misinformation, tie-breaking ratings matched those of human raters more often than LLMs—52% versus 43% respectively. The loss in accuracy here is due to our LLM rater overindexing on any form of misinformation (particularly COVID-19, but also conspiracy theories) which fell outside the stated policy. In other cases, our LLM rater missed a shorthand of misinformation like “80 million dead people,” a census-related claim that more people voted in 2020 than was possible. For violent extremism, tie-breaking ratings most often concurred with human raters (or lacked context), in part due to the policy more cautiously flagging any mention of terrorist groups, whereas the LLM appeared to apply more of a contextual lens.

Taken as a whole, our qualitative analysis suggests that pairing LLM raters with human raters can measurably improve the accuracy of decision making. It also demonstrates that our LLM raters may be more accurate than our ground truth indicates. In practice, queue operators can configure the number of tie-breaking ratings using a higher precision score threshold.

An LLM rater assistant aims to provide critical context to human raters as they render verdicts. As our dataset and analysis is retroactive, we defer evaluation of this design pattern to our real-world experiments.

LLM raters can optimize available human expertise to a scale that was previously cost- and toil-prohibitive. While the current price per inference makes LLM raters best suited for rating content that is sent to existing human rater queues, in the future, LLM costs are likely to decrease. This points to a possibility of LLM raters acting as scaled classifiers. Such an architecture would minimize the training data required compared to existing classifiers, with LLM raters requiring only tens of samples for fixed few-shot prompts, hundreds of samples for adaptive few-shot prompts, or thousands of samples for fine-tuning.

LLM raters enable policy experts to minimize the time between when a threat emerges and when a platform deploys a protection. Policy experts can distill their requirements into a simple set of instructions that an LLM rater can immediately enforce. This agility in adapting policy language—where policy experts rather than engineers can directly iterate on prompts—is critical to responding to the global landscape of user concerns, threat actors, and regulations. Paired with other scaled defense-in-depth systems, LLM raters allow for upstream detectors to be less accurate (and thus easier to develop and deploy), with LLM and human raters correcting for any errors before any enforcement action.

Evolving policies, different cultural norms, and different levels of expertise mean that two human raters can arrive at different verdicts. This is particularly true when human raters are in the midst of adapting their processes to policy changes. LLM raters allow for more consistent—and potentially unbiased—enforcement, acting as a safety net.

LLM raters offer an opportunity to automatically identify the most egregious harmful content, removing the need for human rating. While human raters would still be involved in borderline cases, this shift in balance may improve the well-being of human raters [50].

Similar to traditional security classifiers, LLM raters are potentially susceptible to evasion, particularly via prompt injection [42, 1]. Addressing this requires future research into separating trusted and untrusted inputs and their ability to influence LLM decision making. Alternatively, LLM raters may miss short passages of violative content intentionally padded with otherwise benign content. Addressing this requires either segmenting passages or improving the attention mechanism of LLM raters.

As new LLMs are regularly launched, prompts previously optimized for one version of a model may degrade in performance with the release of a new version. Likewise, optimizations for one LLM may not generalize to another LLM due to idiosyncrasies in RLHF tuning. Safeguarding against regressions requires maintaining both a prompt and evaluation dataset to benchmark changes.

The foundational models underpinning LLMs require massive compute resources which can lead to concept lag where an LLM is unaware of recent events. This poses a challenge for emerging threats—like a violent event or scam tactic—that may not appear in a foundational model’s training data. This risk can potentially be mitigated via our dynamic few-shot example approach, supplying models with emerging context via examples. Related, LLMs may be affected by stereotype or biases [39] which can interfere with the detection of hate speech or harassment. Addressing this risk requires further improvements to debiasing LLMs [18, 26, 33].

An inherent complexity with LLM raters is that the cost per inference depends on the size of the prompt and the size of the LLM. Our experiments demonstrate that platforms have a variety of ways to prioritize cost vs. accuracy. Strategies include using smaller, less expensive LLMs (e.g., text-bison vs. text-unicorn); simplifying policies via automatic prompt optimization [43, 56]; or optimizing the number of examples in a prompt or even the candidate selection strategy [29]. We envision that real-world deployments will consist of distinct layers of LLM raters each with supporting RAG databases. For instance, a platform may first query a fast-path, zero-shot, multi-policy LLM rater with high specificity and recall. Potential violations would then fan out to multiple, specialized, single-policy prompts with adaptive few-shot examples running on a larger LLM model, where any positive verdict from this final layer would result in an escalation.

While our work focused on removing harmful content from platforms, in practice, the ease of configuring prompts and score thresholds means that LLM raters can be equally useful tools for community moderation or even personalized content filters. The need for such tools stems from differing opinions on what constitutes harmful content, particularly in the context of hate and harassment [21, 15, 25].

Making forward progress as a research community towards developing LLM raters requires a canonical “safety benchmark”. Such a dataset would encompass a multi-lingual, multi-modal testing corpus that spans a variety of content safety concepts (e.g., scams, hate, misinformation, and more). Such a benchmark would provide a universal metric for judging progress of LLMs and prompts towards improving the state of user safety online.

Given the fluency of our researcher team and our focus on feasibility, our initial study used only comments in English. Further testing is needed for a wider selection of languages to understanding the performance of LLM raters for a global context.