[SoK] The Challenges of Machine Learning for Trust and Safety:
A Case Study on Misinformation Detection

Abstract

We examine the disconnect between scholarship and practice in applying machine learning to trust and safety problems, using misinformation detection as a case study. We survey literature on automated detection of misinformation across a corpus of 248 well-cited papers in the field. We then examine subsets of papers for data and code availability, design missteps, reproducibility, and generalizability. Our paper corpus includes published work in security, natural language processing, and computational social science. Across these disparate disciplines, we identify common errors in dataset and method design. In general, detection tasks are often meaningfully distinct from the challenges that online services actually face. Datasets and model evaluation are often non-representative of real-world contexts, and evaluation frequently is not independent of model training. We demonstrate the limitations of current detection methods in a series of three representative replication studies. Based on the results of these analyses and our literature survey, we conclude that the current state-of-the-art in fully-automated misinformation detection has limited efficacy in detecting human-generated misinformation. We offer recommendations for evaluating applications of machine learning to trust and safety problems and recommend future directions for research.

1 INTRODUCTION

Online services face a daunting task: There is an unceasing deluge of user-generated content, on the order of hundreds of thousands of posts per minute on popular social media platforms [275]. Some of that content is false, hateful, harassing, extremist, or otherwise problematic. How can platforms reliably and proactively identify these “trust and safety” issues?

Machine learning has proved an attractive approach in the academic literature, leading to large bodies of scholarship on misinformation detection [154], toxic speech classification [357], and other core trust and safety challenges (e.g., [358]). The conceptual appeal of machine learning is that it could address the massive scale of user-generated content on large platforms and the capacity constraints of small platforms. Recent work claims impressive performance statistics: In the literature review that we conduct for this work, among publications that report performance metrics, about 70% of papers report over 80% accuracy on at least one detection task; some of these works report near-perfect performance [43, 44].

In the past year, news items from major tech companies have tempered these expectations. In June of 2023, OpenAI announced the deployment of a content moderation system based on GPT-4 to detect problematic content online; in the same press release, OpenAI’s head of safety systems admitted that human advisors would still need to “adjudicate borderline cases” [355]. In October of 2023, in a Bluesky post commenting on Twitter’s user-driven Community Notes program, Twitter’s former head of trust and safety stated that large-scale automated detection of misinformation remains a hard problem, and that no generalizable automated solutions are currently available [380]. These disclosures accord with our observation that, in practice, trust and safety functions at online services remain heavily manual: driven by user reports and carried out by human moderators.

In this work, we investigate the disconnect between scholarship and practice in applications of machine learning to trust and safety problems. Our project is inspired by recent research that has identified shortcomings in machine learning applications for many problem domains, including information security [331, 336]. We use misinformation detection as a case study for trust and safety problems because the topic has recently generated a rich literature with diverse methods and claimed successes. Misinformation detection has substantive complexities that are common for trust and safety problems: linguistic and cultural nuance, sensitivity to context, and rapidly evolving circumstances.

We seek to answer four discrete research questions, which collectively shed light on the research-practice gap in automated misinformation detection.

• How well-suited are misinformation detection methods in the academic literature to the needs of online services, specifically social media platforms that host user-generated content?

• Are there identifiable missteps related to target selection, dataset curation, feature selection, and evaluations of method performance in ML-driven misinformation detection studies?

• How reproducible are published ML-driven misinformation detection methods?

• How generalizable are published ML-driven misinformation detection methods to out-of-domain data (i.e., to data types and topics not present in training data)?
  

We address these research questions in three ways. First, we conduct a broad literature review and synthesis of the full paper corpus (248 papers), with a focus on detection targets and evaluation. We provide an in-depth review of a subset (87 papers) of the full paper set, with a focus on methods: dataset design, feature engineering, and model selection. Second, we attempt to obtain code and data for a subset of prior work. Third, we test several representative approaches for replication and generalizability. We arrive at the following results by applying these methods.

1. Detection tasks in scholarship are often steps removed from the misinformation content moderation challenges that platforms face. Detection targets may be of limited consequence, or may be more readily and accurately identified through manual means.

2. Methods frequently target proxies for the presence of misleading content; these approaches are easy to evade. Datasets used in publications are often non-representative of real-world contexts. Model evaluation often lacks independence from training and rarely involves close emulation of a real-world deployment.

3. Data and code availability problems pervade the literature, inhibiting replication. Where these are available, we are generally able to replicate prior work.

4. Prior work has poor generalizability when classifying content beyond what was included in training data.

Through this work, we hope to underscore 1) the prevalence and severity of reproducibility and ML-driven method design issues in the existing misinformation detection literature, 2) the need for careful and preemptive evaluation of ML-driven methods at the point of problem formulation, and 3) the importance of method explainability and data accessibility. Based on these observations, we provide recommendations for future work that proposes ML models to address trust and safety concerns. We contribute information taxonomies, our annotated corpus of 248 research papers, recent datasets, and frameworks for evaluating ML-driven content moderation tasks.

2 MOTIVATION

The detection of misinformation and “influence operations” (IOs) is a topic of convergent interest for security, social science, and AI/ML researchers. Methods developed in recent years for the detection of influence operations (e.g., astroturfing, coordinated misinformation campaigns) resemble techniques previously employed by security researchers for the detection of botnets, advanced persistent threats (APTs), and malware [378, 381, 359]. As such, we believe that the security community is uniquely well-positioned to shepherd the responsible development of misinformation detection methods. In addition, misinformation research stands to benefit a good deal from the formal structure of security research methodology: In a survey study from 2022 conducted by Mirza et al., human fact-checkers and journalists expressed interest in adopting rigorous threat modeling practices similar to those found in academic security literature [360].

3 PRELIMINARIES

Definitions. In the absence of agreed-upon definitions of misinformation, disinformation, malinformation, and influence operations, we refrain from advancing singular definitions of the same. Instead, for each paper we review, we consider the paper authors’ working definitions of these terms (if such definitions are provided) and evaluate model performance with respect to the definitions set forth [258, 26, 25, 27, 23]. We limit our analysis to text-based English-language misinformation. We note that disinformation is commonly used to refer to incorrect information written with the intent to deceive, while misinformation refers to incorrect information in general; for the sake of completeness and concision, we refer to intentionally and unintentionally false information as misinformation for this work unless otherwise qualified. We occasionally use false rumors, false news, and false information interchangeably, but avoid fake news, a politically charged term [424]. We use “influence operations” (IOs) or “coordinated campaigns” in our discussion of collaborative attempts to disseminate misinformation across networks [425].

Feasibility of detection. Most of the work we review presupposes that automated detection of misinformation is possible at all. We note that this is, in itself, a strong assumption to make: Published work in psychology, linguistics, and philosophy of language has previously called into question the feasibility of using semantic and syntactic features to determine the veracity of text statements and the usefulness of binary true/false classifications of information, particularly in the absence of unified definitions of misinformation, disinformation, and information [354, 201]. On the other hand, some researchers maintain that misinformative texts are characterized by indelible “fingerprints” that distinguish them from non-misinformative texts: for instance, emotional language and reduced lexical diversity [177, 327]. We use this debate to motivate our discussion of non-textual feature sets for misinformation detection.

Taxonomies. Through an iterative process of reading and inductive coding of our papers, we develop a taxonomy of five “information scopes.” As we read each paper in our corpus, we noted the operative unit of misinformation detection for the classifier or method described—the smallest semantic or organizational unit of information that the method attempted to classify as true or false—and sorted each paper into emergent categories. Our final taxonomy comprises five information scopes: claims, news articles, social media accounts, networks, and websites. We consider these scopes in the context of online services—for instance, news articles and websites whose links might be posted to a social media platform.

• Claims. The smallest semantic unit of fact or misinformation, comprising a subject, predicate, and object, at minimum.

• Articles. News-oriented writing of length 100 words or more.

• Users. All data and metadata associated with a single user’s account as defined by a social media platform (e.g., an account corresponding to a handle on Twitter/X; or a page or profile corresponding to one business, individual, or organization on Facebook).

• Networks. The set of users and interactions between these users as represented by a networked graph of behaviors.

• Websites. A news site, including its hosting infrastructure and text and image contents.

We describe errors in method design with respect to the step of the development pipeline in which they occur. We break this process down as follows: target selection, dataset curation, model choice, feature set selection and model evaluation. Development step definitions follow:

• Target selection. The stated versus actual objective(s) of the classification task that paper authors describe: for instance, detection of emotional valence of a text (angry, sad, happy); verification of semantic accuracy (true, false); or characterizing degree of virality (e.g., as measured by number of reshares on a post).

• Dataset curation. The source, size, and contents of datasets used for model training and testing. Provenance information includes temporal labels for 1) the date of the dataset’s production and 2) the date of dataset access by paper authors.

• Model choice. The choice of ML model used by paper authors for their detection task, as well as their motivation for this choice.

• Feature selection. The choice of feature(s) that paper authors use to train their ML models, as well as their motivation for this choice.

• Model evaluation. Paper authors’ approach to benchmarking method performance after initial training. This includes 1) choice of test dataset; 2) performance statistics (e.g., ROC values, true and false positive rates); and 3) ecological validity of test cases in relation to proposed deployment setting.

Paper organization. In Section 2, we situate this project within existing security literature and academic literature that critiques machine learning-driven methods. We motivate our choice of automated misinformation detection as a representative case study and highlight the particular relevance of this problem to the security community. In Section 3, we present the information taxonomies developed from our inductive coding of papers. We discuss findings from our reading and coding of papers in Section 4. Our systematization of literature progresses along two axes: 1) from unimodal to increasingly multi-modal methods, and 2) in order of operations of method development. Specifically, we discuss issues pertaining to method-target fit, dataset curation and feature selection, model selection, and method evaluation (RQ1, RQ2). In Section 5, we illustrate the issues identified in our literature review with a series of replication studies, with a focus on reproducibility and generalizability of results (RQ3, RQ4). In Section 6, we conclude with 1) a discussion of findings from our literature review and replication studies and 2) generalizable recommendations for evaluating ML-driven interventions for trust and safety.

4 Systematization of Literature

Paper selection. To seed our corpus, we manually curated a selection of 23 highly-cited survey papers that provide comprehensive overviews of the state of automated misinformation detection at the time of writing [151, 152, 153, 154, 155, 156, 157, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174]. We relied on these papers to ground our own understanding of existing approaches to automated misinformation detection, and to identify detection methods that have been well-received by the research community. We searched for survey papers in Google Scholar with queries “survey misinformation detection” and “survey fake news detection” and collected the most-frequently cited papers within the past 10 years. We then inspected each paper’s reference section for related papers; we read the abstracts for these references in order to confirm relevance and fit. We supplemented this core corpus of highly-cited papers with publications surfaced by Google Scholar queries. We queried the following terms on Google Scholar: “misinformation detection [$x$]” and “automated fact checking [$x$],” where $x$ $\in\{$claims, news articles, accounts, networks, websites, influence operations$\}$. We collected the 50 most highly-cited papers in the set resulting from the union of search results returned by both search queries for each $x$. To counter potential bias toward older publications, we collected papers with the highest citation rates per year. We note that these search terms are deliberately broad and over-inclusive; we manually review all potential works for relevance after the initial sampling step. After removal of out-of-scope works (see In- and out-of-scope work.) from this initial set of 250 papers, 219 eligible papers remained.

To ensure that security-oriented approaches to detection were represented in our corpus, we conducted a separate snowball sampling search for work published in security venues: Using the same keywords listed in the previous subsection, we oversampled publications from four prominent security research venues (USENIX Security, IEEE S&P, NDSS, and ACM CCS). This sampling process resulted in the addition of 29 additional works, most of which address the detection of accounts and networks that spread misinformative content (e.g., botnets and trolls); as such, most of these works are categorized as network- or account-scoped detection methods. Our final corpus comprises 248 papers published between 2009 and 2023, inclusive.

“Full” and “focus” paper corpora. We conduct our literature survey at two different levels of granularity. For all papers in the full corpus (248 works), we note detection targets and model evaluation (steps and in Section 3). For a subset of these (87), we perform deep-coding of methods: we note dataset design choices, model selection, and feature sets (resp. steps , , and in Section 3). A summary of this deep-coding is available in Table I. Our motivation for developing this focus set is practical: A good many of the works we review do not include in-depth discussions of model choice, weights, and datasets. As such, each per-scope discussion is book-ended by comprehensive overviews of the full paper corpus, with more focused analyses of method design details in between.

Corpus curation. As a first-pass relevancy check, we used automated keyword matching to confirm that all collected papers did, in fact, address misinformation and automated fact-checking methods; we read the abstracts of papers surfaced by this check to confirm relevance and fit. We then annotated research papers that passed this check in accordance with our codebook (see SI). One reader made three separate coding passes over the corpus, varying the objectives of her annotation on each pass: In the first pass, she specifically sought to identify taxonomies for classification and detection; on the second, she noted actual versus stated detection targets and approaches to evaluation; on the final pass, she noted method design approaches and errors.

In- and out-of-scope work. We consider a number of security-oriented approaches to misinformation detection in this work. Sybil and botnet detection methods that specifically target influence operations online are categorized as account- and network-scoped detection methods in our corpus. Commercial approaches to misinformation detection are in-scope for this project. In view of data accessibility issues, however, we are unable to provide an in-depth analysis of commercial methodologies in our main literature review, and instead include a market survey of commercial fact-checking providers in our online supplement. (In general, commercial vendors do not make code and training data publicly available.) LLM-powered detection is in-scope for this work. Accessibility to code bases for LLMs is similarly limited (this has been discussed at length in the popular press [393, 394]), and disallows extensive testing and evaluation by researchers. As such, while we do briefly discuss LLM-powered detection approaches in our supplement, we defer a more extensive discussion to future work.

Notation. In these sections, we denote information scope and pipeline steps with circled icons ( , ) and subtaxa within those categories with lowercase bolded Roman numerals (iv). We note that, while targets are generally unique to each information scope (and are referenced by an alphanumeric label (e.g., (A.i)), critiques of steps - are not scope-specific, and are always denoted by Arabic-Roman numeral pairs (e.g., (2.i)). We highlight each subtaxon in-text, with a different color corresponding to each development step. (The full taxonomy is in Section 7.) A summary of findings for each scope is available in the Takeaways box at the end of each subsection.

Paper		Target	Dataset	Model	Features				Performance
Work	Scope				Textual	Network-based	Author-based	Infrastructural	Accuracy/AUROC
1. Ajao et al. [278]		Sentiment (A.i)	PHEME [146]	LSTM, DT, RF, SVM					0.86 (Acc.)
2. Abulldah-All-Tanvir et al. [210]		Content (C.i)	Twitter (API)	NB, RNN, LSTM, SVM, Logit					0.89 (Acc.)
3. Bhutani et al. [236]		Content (C.i); sentiment (A.i)	Twitter (API), PolitiFact [299]	Naive Bayes, RF					0.60 (AUC)
4. Bozarth et al. [20]		Contents (C.i)	PolitiFact [299], Daily Dot, Zimdars, MBFC	LDA					n/a
5. Ciampaglia et al. [22]		Shortest path search (C.ii)	DBpedia	kNN, RF					0.97 (AUC)
6. Cui et al. [277]		Content (C.i); sentiment (A.i); user response (U.ii)	PolitiFact[299], GossipCop[343]	KNN, SVM, CSI [37], RMSprop					0.82 (F1)
7. Debnath et al. [84]		Content (C.i)	LIAR [35]	CNN					0.27 (Acc.)
8. Dey et al. [241]		Content (C.i); sentiment (A.i)	Twitter (API)	Clustering (kNN)					0.67 (Acc.)
9. Galitsky et al. [215]		Content (C.i)	Amazon reviews	Parse thicket					0.81 (Prec.)
10. Glockner et al. [142]		Content (C.i)	PolitiFact[299], Snopes[298], MultiFC	CNN, DNN					0.58 (Acc.)
11. Gordon et al. [110]		Content (C.i); source rep (U.i)	Credibility-Factors2020	SVD					0.63 (Acc.)
12. Gupta et al. [294]		Content (C.i); stance (A.iii)	Twitter (API)	SVM					0.60 (Agreement)
13. Hassan et al. [130]		Content (C.i); checkability (C.iii)	NBA, weather datasets	Frequency					n/a
14. Jain et al. [291]		Content (C.i); sentiment (A.i)	Twitter (API)	Gensim/TextBlob					0.77 (Acc.)
15. Jiang et al. [32]		Content (C.i); ling./syntax (C.ii)	PolitiFact[299], Snopes[298]	SVM					0.81 (Acc.)
16. Karimi et al. [260]		Content (C.i)	LIAR[35]	LSTM, CNN					0.39 (Acc.)
17. Kartal et al. [108]		Content (C.i); checkability (C.iii)	Check That! dataset	Logit, SVM, RF					0.26 (MAP)
18. Kou et al. [128]		Content (C.i)	CoAID, CONSTRAINT	Knowledge graph					0.90 (Acc.)
19. Paudel et al. [185]		Keyword detection (A.ii)	Abilov et al. dataset, Twitter (API)	AdaRank, ListNet, RF					0.79 (MAP)
20. Popat et al. [90]		Content (C.i)	PolitiFact[299], Snopes[298], NewsTrust	biLSTM, CNN					0.88 (AUC)
21. Shiralkar et al. [56]		KG search (C.ii)	DBpedia	Knowledge graph					1.00 (AUC)
22. Shu et al. [207]		Word encoding (C.i); user response (U.ii)	GossipCop [343], PolitiFact [299]	RNN/RMSprop, CSI [37], LSTM, CNN					0.93 (F1)
23. Tian et al. [106]		Content (C.i); user response (U.ii)	Twitter15, Twitter16	CNN-biLSTM					0.82 (F1)
24. Zhang et al. [382]		Content (C.i); user response (U.ii)	RumourEval, PHEME[146]	biLSTM, Multitask, SVM, CNN,					0.89 (Acc.)
1. Afroz et al. [212]		Content (C.i); syntax (A.i)	Brennan-Greenstadt	SVM, J48 Decision Trees					0.97 (F1)
2. Ahmed et al. [214]		Syntax (A.i)	Twitter, Kaggle, Horne and Adali [24]	SVM					0.92 (Acc.)
3. Bourgonje et al. [55]		Stance (A.ii)	Fake News Challenge Data	Logit					0.90 (Acc.)
4. Brasoveanu et al. [68]		Sentiment (A.i); keywords (A.ii)	LIAR[35]	CNN, LSTM, CN					0.64 (Acc.)
5. Della Vedova et al. [58]		Content (C.i); virality (N.iv)	FakeNewsNet, Buzzfeed	Logit					0.82 (Acc.)
6. Horne et al. [24]		Syntax (A.i), headline (A.ii)	Buzzfeed, Burfoot & Baldwin	SVM					0.78 (Acc.)
7. Jabiyev et al. [118]		Topic detection (A.ii); site cred. (W.iv)	Snopes[298], FactCheck, PolitiFact[299]	SVM, DT, RF					0.87 (Acc.)
8. Jadhav et al. [238]		Content (C.i); syntax (A.i)	LIAR [35]	DSSM/RNN					0.99 (Acc.)
9. Jin et al. [216]		(A.ii); (N.i); suspicious accounts (U.i)	Tweets; articles	n/a					0.87 (Prec.)
10. Kapusta et al. [88]		Sentiment & word freq. (A.i)	MBFC and custom	n/a					n/a
11. Kumar et al. [29]		(A.i); UI (W.iii); Author cred. (U.i)	20K Wiki Hoaxes	Random forest					0.87 (AUC)
12. Magdy et al. [18]		Content (C.i)	NYT Corpus [224], 100 Wikis	Pattern recog.					0.99 (Recall)
13. Monti et al. [315]		(A.ii); (N.i); suspicious accounts (U.i)	Tweets; articles	RNN/CNN					0.927 (AUC)
14. Nasir et al. [141]		Syntax (A.i)	ISOT [312]; FAKES [313]	RNN/CNN					0.99 (Acc.)
15. Perez-Rosas et al. [63]		Syntax (A.i)	FakeNewsAMT; Celebrity	SVM					0.74 (Acc.)
16. Potthast et al. [17]		Syntax, sentiment, readability (A.i)	Buzzfeed-Webis	Bag-of-words					0.46 (F1)
17. Reis et al. [286]		Syntax (A.i); contents (C.i); source rep. (U.i); timing (N.ii); URL (W.i)	BuzzFeed	GBM					0.85 (AUC)
18. Rubin et al. [292]		Syntax (A.i)	AMT	Clustering					0.67 (Agreement)
19. Ruchansky et al. [37]		(A.i); Responses (A.ii); acct metadata (U.i)	Twitter/Weibo posts	RNN/LSTM					0.95 (Acc.)
20. Santos et al. [19]		Readability (A.i)	Fake.Br corpus	SVM					0.92 (Acc.)
21. Silva et al. [111]		Topic detection (A.ii); propagation (N.i)	PolitiFact[299], GossipCop[343], CoAID	Clustering					0.88 (Acc.)
21. Singh et al. [203]		Syntax (A.i)	Kaggle Fake News	SVM					0.87 (Acc.)
22. Uppal et al. [102]		Discourse structure (A.i)	Buzzfeed, PolitiFact[299]	GRU, Dependency tree					0.74 (Acc.)
1. Cao et al. [101]		Acct. cred. (U.i); prop. (N.i)	Tuenti social network	Louvain clustering					0.90+ (TP)
2. Danezis et al. [373]		Acct. cred. (U.i); prop. (N.i)	LiveJournal data	Bayesian inf.					n/a*
3. Ezzeddine et al. [366]		Acct. behaviors (U.ii)	DATA	LSTM					0.91 (AUC)
4. Hamdi et al. [91]		Account metadata (U.i); prop. (N.i)	CREDBANK	LDA, Bayes, Logit, SVM					0.99 (AUC)
5. Helmstetter et al. [5]		Acct metadata (U.i); post sharing data (U.ii)	Public site cred. lists	SVM, NB, DT, RF					0.936 (F1)
6. Jain et al. [291]		Acct metadata (U.i); topic det. (A.ii)	Twitter (API)	Gensim, TextBlob					0.77 (Acc.)
7. Leonardi et al. [38]		Acct metadata (U.i); prop. (N.i)	CoAID	RF					0.81 (F1)
8. Saeed et al. [361]		User behavior (U.ii)	Reddit Pushshift; Reddit IRA trolls list	RF					0.98 (Acc.)
9. Sansonetti et al. [384]		Acct metadata (U.i); acct activity (U.ii)	PolitiFact [299], Twitter (API)	LSTM-CNN, SVM, KNN					0.92 (Acc.)
10. Santia et al. [21]		Source rep. (U.i); user response (U.ii); syntax (A.i)	BuzzFeed	SVM, RF, DT, NB					0.77 (Prec.)
11. Shu et al. [226]		User behavior (U.ii); prop. (N.i) syntax (A.i)	Buzzfeed, PolitiFact	Gibbs sampling					0.85+ (Acc.)
12. Vargas et al. [376]		Prop. (N.i); topic det. (A.ii)	Twitter (API)	RF					0.98 (F1)
13. Wang et al. [191]		User behavior (U.ii)	Renren data	SVM					0.99 (Acc.)
14. Yu et al. [200]		User behavior (U.i); prop. (N.i)	LiveJournal, Friendster, DBLP accounts	Random route					n/a*
15. Yuan et al. [375]		Acct metadata (U.i); timing (N.ii)	WeChat data	Clustering					0.90+ (Prec.)
16. Zhang et al. [257]		User behavior (U.ii); prop. (N.i)	Twitter, Slashdot, Epinion	Graph cut					n/a
17. Zhou et al. [72]		User suscept. (U.i); prop (N.i)	PolitiFact [299], BuzzFeed	SVM, KNN, NB, DT, RF					0.93 (Acc.)
1. Alizadeh et al. [370]		Propagation (N.i); syntax (A.i); acct metadata (U.i)	Twitter (API), Reddit IRA troll list	RF					0.70+ (F1)
2. Antoniadis et al. [178]		Acct metadata (U.i); syntax (A.i)	Hurricane Sandy tweet dataset	J48, RF, KNN, Bayes					0.79 (Avg. Prec.)
3. Assenmacher et al. [391]		Propagation (N.i); topic det. (A.ii)	Twitter (API)	Clustering					not reported
4. Buntain et al. [50]		Time (N.ii); acct metadata (U.i); sentiment (A.i)	CREDBANK, Buzzfeed	RF					0.65 (Acc.)
5. Castillo et al. [253]		Syntax (A.i); user behavior (U.ii)	Twitter Monitor events	SVM, DT					0.874 (P)
6. Chen et al. [183]		Social graph (N.iii); syntax (A.i); user behavior (U.ii)	Weibo	RNN					0.92 (Acc.)
7. Guo et al. [272]		Prop. (N.i); semantics (C.i)	Twitter, Weibo	LSTM					0.9 (Acc.)
8. Jin et al. [263]		Propagation (N.i); stance (A.iii)	Sina Weibo posts	Clustering					0.84 (Acc.)
9. Liu et al. [40]		Propagation (N.i)	Weibo, Twitter15, Twitter16	RNN, CNN					0.897 (Acc.)
10. Ma et al. [30]		Propagation (N.i)	Kochina, Ma, Shu Twitter datasets	RNN/BiLSTM					0.75 (Acc.)
11. Magelinski et al. [262]		Prop. (N.i); timing (N.ii); user behavior (U.ii)	Twitter (API)	–					– n/a
12. Nguyen et al. [69]		Prop. (N.i); semantics (A.i)	Twitter, Weibo, PHEME [146]	SVM, RNN, DT					0.970 (Acc.)
13. Pacheco et al. [92]		Account metadata (U.i); timing (N.i)	Twitter (API)	Clustering					0.8+ (Prec.)
14. Ratkiewicz et al. [388]		Prop. (N.i); keywords (A.ii); user behavior (U.ii)	Twitter (API)	AdaBoost, SVM					0.96 (Acc.)
15. Sharma et al. [158]		Prop. (N.i); keywords (A.ii); user behavior (U.ii)	Twitter (API); Twitter IRA trolls list	GMM, Kmeans, NN					0.94 (AUC)
16. Tschiatschek et al. [230]		Prop. (N.i); user rep. (U.i)	Facebook dataset	Bayes inf.					n/a
17. Zeng et al. [133]		Prop. (N.i); stance (A.iii)	4.3K tweets (from API)	Logit, NB, RF					0.88 (Acc.)
1. Asr et al. [287]		Source rep. (W.i); Syntax (A.i)	BuzzfeedUSE, Snopes, Rashkin, Rubin	CNN, SVM, NB					not reported
2. Baly et al. [43]		Source rep. (W.i); Site infra. (W.ii); (A.i)	MediaBiasFactCheck[311]	SVM					0.66 (Acc.)
3. Baly et al. [316]		Source rep. (W.i); Site infra. (W.ii); (A.i)	MediaBiasFactCheck[311]	SVM					0.7152 (Acc.)
4. Castelo et al. [302]		Site infra. (W.ii); syntax (A.i)	Celebrity, US-Election2016	SVM, kNN, RF					0.86 (Acc.)
5. Chen et al. [109]		Hosting infra. (URL) (W.ii); syntax (A.i)	PoliticalFakeNews	Clustering					0.97 (AUC)
6. Hounsel et al. [1]		Site infra. (W.i)	FactCheck, Snopes, PolitiFact, Buzzfeed	RF					0.98 (AUC)
7. Ribeiro et al. [254]		Site infra. (W.i); User demog. (U.i)	Facebook API	Graph search					0.97 (PCC)

TABLE I: Focus corpus by scope and target. Coding of a focus set of 87 papers, sorted by information scope. Values in parentheses in “Target” field correspond to highlighted subcategories presented in Section 4 (e.g., “C.i” denotes target (i) in “Claims,” Section 4.1). If authors present evaluation results for multiple models, we underline the most performant model and record its corresponding performance score.

4.1 Claims

About 70% of paper authors within this scope cite the 2016 U.S. presidential election as motivation for the development of their methods [84, 130, 297]; about 30% cite COVID-19 misinformation [128, 121, 142]. All works mention the speed and volume of social media misinformation in particular [108]. 30% cite the relatively slow pace of manual fact-checking [22, 130, 108]—and the need for faster, automated approaches—as motivation. Claim-scoped papers form 15% of our corpus.

Detection targets. Across all information scopes, we find that claim-scoped detection methods are most consistent in their attempts to verify semantic contents of text statements. This is done in the following ways: (C.i) distance calculations on semantic embeddings to perform textual entailment or stance detection [77, 48]; and (C.ii) search on a knowledge graph topology [22, 56] to determine if these reference sources corroborate or refute the claim to be checked. A small class of approaches explicitly detect (C.iii) “checkworthiness,”  and are intended to surface checkable statements to human fact-checkers for manual verification [108, 56, 130]. Targets such as author credibility and language cues are proxy targets for the presence of misinformation: signals which may not be sufficient for determining text veracity in isolation, but which are indicative of (lack of) veracity by association with an external heuristic. About 60% of papers within the full corpus at this scope employ non-semantic targets, including the (A.i) syntactic and/or stylometric qualities of text [238], (U.i) source reputation [383], or (U.ii) contextual indicators, such as commenter responses [106], to classify social media posts.

Datasets. Claim-scoped papers in our corpus that propose to perform fact verification (2.i) rely on outdated existing datasets of labeled statements in order to establish ground truth. LIAR [35] and PolitiFact [299] were the most popular datasets among claim-scoped works, and, taken together, were used by approximately half of all papers within this scope [76, 123]. LIAR is a static political news dataset that was published in 2017; on average, papers that cite LIAR were published two years after LIAR’s release. Topic detection and word frequency models trained on LIAR are likely ineffective in contemporary fact-checking contexts, and for non-political subject matter [84, 383]. Additionally, misinformation taxonomies across reference sites are inconsistent: PolitiFact employs a six-point labeling scale (pants-on-fire; false; barely-true; half-true; mostly-true; true), FEVER employs a three-point scale (supported; refuted; notenoughinfo), and GossipCop employs an eleven-point scale (ratings from 0 to 10). This divergence is likely a symptom of definitional issues (Section 3).

Model selection. Model choice follows target choice for claim-scoped methods: for methods that pre-construct knowledge graphs or other reference databases, models perform some form of (3.i) shortest-path search on the KG topology [56, 22], and approximate logical inference via transitive closure on graph edges. For methods that perform information retrieval at query time—e.g., to match corroborating sources to a claim to be checked—model training generally follows conversion of the text statement to a bag-of-words or TF-IDF embedding; choice of model is highly variable [236, 207], and does not appear to be predictive of performance. For methods that perform detection of misinformative posts on social media, (3.v) stacked ensemble classifiers are a common approach to incorporating multiple feature modalities.

Feature selection. At the level of single claims, semantic feature analysis is limited to the (4.i) identification of structured statements as a precursor to knowledge graph (KG) construction. These claims take the form of subject-predicate-object (SPO) statements (e.g., “I like pie”) [22, 18, 56]. Detection versatility is determined by the size of the source dataset and the granularity of the relationships encoded by graph edges [56]. Supervised methods that detect linguistic cues employ (4.i) hand-crafted word or topic lists [108]). Authors employing supervised methods claim that their approach permits highly customized targeting of specific rumoring narratives [108], though this also assumes that the method developer has prior knowledge of the contents of test data; authors employing unsupervised methods claim that their approaches detect contextual and language features that cannot be easily extracted by common features such as word frequency or sentiment [76]. Methods detecting social media data consider (4.ii) network and (4.iii) user interaction features (post likes, shares, comments) [185, 294, 277].

Evaluation. Though a majority of claim-scoped methods cite the speed of social media misinformation as motivation, (5.i) only one method within this scope reported results from testing in real time [54]. KG-based methods are (5.ii) non-generalizable by design: the approaches we survey require structured inputs for graph construction, and rely on published datasets for ground truth [22, 56]. Though this approach permits semantic verification of statements, it is difficult to perform iterative updates to source databases in real-time, particularly in the types of online settings where claim-checking might be most usefully deployed (e.g., during breaking news events, where no source of ground truth is immediately available). Additionally, we observe that a majority of works at this scope do not test on novel or out-of-domain data; we discuss overfitting issues in greater detail in the next section.

4.2 Articles

We consider all news-oriented writing of length 100 words or greater to fall within this scope. 25% of papers within this scope cite growing distrust of mainstream news media outlets as motivation for their methods, which promise to deliver fast labeling of news stories that appear on social media [174]. Text-based credibility classifiers have been shown to have limited efficacy, however: while unsupervised approaches can identify bias with high accuracy, this performance degrades significantly in misinformation and credibility classification tasks [395, 17]. This scope comprises 24% of our full paper corpus.

Detection targets. In contrast to single claims, full-length news articles have sufficient text contents to make semantic verification difficult—and certain off-the-shelf NLP approaches practicable. These approaches are distinct from direct claim verification and qualify as proxy detection methods: For instance, Bhutani et al. associate strongly negative sentiment with the presence of false information [236], and Horne et al. find that satire and misinformation share stylistic similarities [24]. All article-scoped methods target proxy signals, and adopt at least one of the following three approaches to detection: (A.i) NLP analysis of article contents to identify language features particular to writing styles heuristically associated with misinformation (genre detection, sentiment analysis) [324]; and (A.ii) analysis of article contents and headlines to identify potentially clickbait-y titles or discussion related to known misinformation narratives (topic detection) [163, 108]. Respectively, these approaches 1) simultaneously assume and detect a heuristic (e.g., strong emotion indicating the presence of misinformation); and 2) assume prior knowledge of rumoring topics.

Datasets. While well-annotated, current datasets are in short supply across all information scopes, this deficit is particularly glaring at the article scope. This is due in large part to definitional ambiguities that prevent fine-grained labeling of longer texts for classifier training. As a result, 44% of paper authors within this scope (2.i) use public datasets released years prior to the start of their research [68, 35, 24, 41]. These datasets (LIAR [35], Buzzfeed-Webis [17], and PolitiFact [299]) include news links, speaker credibility scores, and other metadata that (2.ii) constitute serious sources of leakage for methods that use contextual features to infer true/false labels. The remaining 56% of authors curate their own article corpora by asking crowdworkers to generate misinformative text [63], selectively editing true news articles (e.g., via verb inversion or noun replacement) [31], or compiling articles from authoritative news sources and known satire sites [240]. These data curation techniques (2.iii) introduce additional dependencies and shortcuts to textual datasets for which such variables are already difficult to control. Style [17] and genre [89], for instance, are emergent qualities of writing that cannot be easily marginalized out of a text embedding.

Model selection. Among papers that report testing with multiple models—including (3.ii) classical ML models [214], (3.iii) unsupervised NN models [68], and (3.v) stacked ensemble classifiers [37]—there is no clear correlation between model choice and actual performance. We note that, in instances where authors test on two- and multi (i.e., $>$ 2)-way classification tasks, performance declines sharply in the latter case [212]. In those instances, reported performance scores are for two-way tasks. For this reason, as well, classical ML models (logit, SVM) oftentimes appear to be most performant.

Feature selection. Among supervised methods that disclose their feature sets, we find that (4.i) word frequency, sentiment, and genre were among the most commonly used features, and were collectively employed by 80% of works within this scope; these features can also be sources of dependency-induced noise. It is difficult to quantify the impact of dependencies related to voice, house style, and source on classifier performance, particularly in the case of unsupervised learning methods, which comprise 59% of methods at this scope. We evaluate an unsupervised learning method (and consider its performance in light of possible style-related dependencies) in our replication analysis of Nasir et al. (Section 5.1) [141].

Evaluation. Misinformation detection methods scoped to full texts risk overfitting to single topics: 42% of authors select (5.ii) one or more narratives of interest (e.g., the 2016 presidential election, the Boston Marathon bombing), train a classifier on these topics, then test this classifier on a different set of texts that discuss the same topic [276, 216, 141]. This approach, while valid for evaluating classifier performance on closed datasets, lacks ecological validity for the use cases that authors claim that their methods will address: Rapid topic identification and high-quality annotation of relevant articles are generally unavailable in breaking news scenarios on social media platforms [396]. 60% of authors at this scope compare the performance of their detection method to other published approaches or ML models, but (5.ii) neglect to test on novel datasets. These methods do well when tested on in-domain texts, and in comparison to a selection of older ML models; many report accuracy well above 80% [73, 111]. Only one paper within the article scope tested in an adversarial setting: Its authors found that, while stylometry-based misinformation detection had an accuracy rate greater than 80% on routine tasks, this score dropped to about 50% in adversarial cases [212]. We demonstrate such a performance dropoff in our replication analysis (Section 5.1).

4.3 Users

Evidence of foreign interference during the 2016 U.S. presidential election triggered a resurgent interest in malicious account detection [366, 361]. As such, 90% of security- and social-science-oriented works that we include within this scope (a dozen papers) explicitly discuss Russian trolls or other influence operations conducted by nation state actors and train classifiers on published lists of such accounts [361, 362, 360, 364, 365]. Account-scoped papers formed approximately 15% of our corpus (40 papers).

Detection targets. Papers within this scope target source reputation, and (U.i) inspect account metadata, such bios, account age, and profile images; or distinguish suspicious accounts by (U.ii) a single user’s social behaviors, such as their comments on posts (n.b. this target is distinct from (N.i)). The security literature we review discusses trolls and bots deployed for astroturfing, misinformation campaigns, and IOs [361, 367, 368, 369, 370]. In the absence of rigorous definitions of these account types, however, actual detection targets are tautological: A troll or bot is an account that exhibits troll- or bot-like behavior, or that interacts with confirmed troll or bot accounts [361, 367].

Datasets. All troll and bot account detection works we reviewed relied on published lists of “known” troll accounts for model training, but (2.iv) neglected to mention the heavily manual investigation required to produce these original lists [300, 385]. Researchers who compiled some of these account lists, including a set of several hundred Twitter accounts with possible links to a known Russian troll farm (the Internet Research Agency, or IRA) manually examined suspicious accounts and tweet contents in order to produce detailed account and content taxonomies; notably, these classifications required external intelligence about account activity that was not published alongside account lists [385]. Two other highly-cited troll lists compiled by the U.S. government, comprising several thousand suspicious Twitter and Facebook accounts, were curated through the use of proprietary non-public information [423].

Model selection. Detection proceeds via classifier training on a list of “known” suspicious accounts and application of this classifier to a dataset of novel accounts [361, 364]. We note that, regardless of model choice, if classifier training data and feature selection reflect a heuristic about suspicious behavior, the resulting classifier will simply learn this heuristic: The methods we review can be used to detect accounts whose behaviors conform to heuristic assumptions, but cannot be used to surface novel malicious behaviors, and are not resistant to attacks or evasion [371]; we explore this further in our replication analysis of Saeed et al. (Section 5.2) [361]. A subset of methods at this scope and the network scope formulate detection as a (3.iv) graph cut or influence maximization problem, and describe approaches to identifying optimal cuts for isolating suspicious accounts [257, 101].

Feature selection. Methods that define suspicious accounts by intrinsic properties of these accounts (e.g., user handles and profile images) target the (4.iii) semantics of this account metadata, and detect evidence of manipulation in (e.g.) image metadata and bios; or text outputs, such as posts and links [5, 384]. Methods that define suspicious accounts by account activity target (4.ii) networked behaviors, such as liking and resharing statistics [92]. Feature sets for some methods in the first category include demographic data for users, such as inferred political party affiliation or race [254] (these nth order assumptions are dangerous to make [100]; see our discussion about proxy signals for detection, in Section 4.2.

Evaluation. We observe accuracy scores above 80% for (5.ii) confirmatory detection of like accounts for all methods that reference a seed list of known trolls [361, 66, 366]; de novo detection of behaviors not represented within training data is not possible, by the self-admission of 10% of authors within this scope [361, 359]. As we discuss further in the next subsection (Networks), the increasingly hybrid nature of IOs requires more nuanced taxonomies for classification: extent of coordination, rather than existence, might be a more appropriate measure of possible manipulation. Formerly, Botometer did this by assigning a “bot-ness” score to individual Twitter accounts that gauged extent of automation [422]. Though currently defunct post-Twitter API shutdown, Botometer performance had already degraded significantly on Twitter data from 2020, per Rauchfleisch et al. [372]. Some authors of bot detection methods acknowledge that their approaches are (5.iii) trivially easy to evade if account holders 1) avoid interacting with known suspicious accounts or 2) vary their account identity and posting semantics [373, 367].

4.4 Networks

Within the security literature, a growing awareness of hybrid networks, which employ a combination of automated and manual approaches to disseminate content, has encouraged a turn toward network-based bot detection methods, and away from detection of individual accounts [367, 374]. We observe a parallel turn toward network-based methods in AI, ML, and NLP venues as a result of growing recognition of overfitting and generalizability issues in purely text-based detection methods (see Articles) [302]. The common assumption, across disciplines, is that coordinated networks leave more detectable evidence of manipulation than do individual accounts, and that these footprints should be identifiable regardless of attack type or rumoring topic [302, 378]. Network-scoped methods, including relevant work in the security literature, form 20% of our corpus.

Detection targets. All methods at this scope identify patterns of user interaction and content propagation as targets; these methods associate virality with the existence of rumoring narratives [6, 228, 41] and temporally anomalous activity with evidence of coordination [359, 367, 364]. The corresponding targets for these approaches are (N.i) propagation patterns across social graph topologies and (N.ii) temporal records of user-user interactions. Within non-security misinformation literature, we note that virality assumptions disallow early detection of misinformation [315, 253]. Similarly, within the security literature, anomalous patterns of account registration and user interaction serve as proxies for the presence of Sybils and botnets [375, 373]; early detection requires that authors formulate a priori assumptions about the nature of these patterns.

Datasets. We conducted an author outreach survey for works within this scope in an attempt to locate hard-to-find social media datasets. We found that (2.v) accessibility issues were exacerbated by the shutdown of the Twitter API [308]. In total, we attempted to locate datasets and code for 50 different papers (see Section 5 for our methodology). Thirty-six (72%) of these analyzed tweet corpora, and 42 (84%) of these targeted social media users and posting contents. We were able to independently source complete methods or data for fourteen (28%) of these. Of the 27 authors we eventually contacted about providing partial or dehydrated datasets (as we were unable to locate these datasets ourselves), nine responded; six of those authors were able to provide method code or partial datasets.

Model selection. The network-scoped methods we review formulate detection as 1) a structured content classification problem [370, 388], and/or 2) a clustering problem on social graphs [391, 157]. In the former case, authors employ an assortment of (3.ii, 3.iii) supervised and unsupervised models to detect suspicious language across multiple accounts. In the latter case, authors use (3.vi) Louvain or Kmeans clustering or K-nearest neighbors to detect neighborhoods of suspicious accounts, as determined by user-user interactions. Though methods in the latter category advertise themselves as content-agnostic, we note that published methods [40, 30] access datasets of social media posts that were already sorted by rumoring topic or event [189, 202].

Feature selection. 55% of papers within this scope make normative theoretical assumptions about user behaviors: In keeping with an epidemiological model of misinformation spread [132, 328], Nguyen et al. assume a homogeneous population of newsreaders, with (4.ii) identical probabilities of “infection” and reinfection [205]. Similarly, in the security literature, techniques for detecting bots and Sybils identify behaviors that align with heuristics determined a priori by researchers: These methods assume, for instance, that Sybils will form well-connected neighborhoods [375], or (seemingly contradictorily) that compromised Sybils will refrain from connecting with additional Sybils, to avoid detection [373].

Evaluation. Ferrara et al. [379] called attention to the false positive rate problem in botnet detection in 2016, noting that classifiers for bot detection only work well in instances where there is a clear-cut distinction between bot and non-bot accounts. This distinction is becoming increasingly blurred, however. Sophisticated network-based attacks try to engage non-bot accounts in organic interactions with bot accounts (e.g., astroturfing attacks) [367, 359], (5.v) rendering even positive detection results insufficient or meaningless: coordinated activity need not be inauthentic, and inauthentic activity need not be malicious.

4.5 Websites

Methods within this scope apply credibility, factuality, or (political) bias scores to whole news sites; authors claim that site-wide labels can be used to quickly infer the quality of individual news articles produced by these sites [1, 316, 302, 109]. As with article-scoped methods, we noted intervention fit issues at the whole website scope. Asr et al. found that whole-source labels were insufficient proxies for the truthfulness of single news articles, and elided subject-specific variations in reporting quality [287]. (We discuss dataset distribution in greater depth in Datasets.)

Detection targets. In 50% of works that we review in this scope, authors reduce the task of whole-site credibility labeling to a significantly smaller, unimodal classification task: Chen et al. [109] (W.i) detect suspicious domain semantics (in essence, a text classification task on URLs); Ribeiro et al. [254] (W.ii) infer site bias from site visitor demographics; Castillo et al. [253] (W.iii) detect suspicious ad interfaces and markup features; Baly et al. [43, 316] and Hounsel et al. [1] present methods incorporating (W.iv) infrastructural features, though the overall performance of the method of Baly et al. is strongly determined by performance on the text classification task alone (thus, in practice, the method closely resembles the article-scoped detection methods we examine, and is susceptible to the same dependencies that we observe in that scope). We demonstrate this via an ablation analysis in our replication study of their method (see Section 5.3).

Datasets. For both training and testing, all methods scoped to whole website detection rely on published lists of websites with accompanying credibility scores [43, 316, 1, 302]; common reference sites include Media Bias/Fact Check, Snopes, and FactCheck.org [311, 298, 402]. As discussed in Section 4.1, however, these references do not have uniform taxonomies for classifying site credibility. Additionally, pre-labeled lists are 1) (2.i) biased towards older, more visible real news and fake news outlets [298], 2) (2.iii) are restricted to specific information domains [43], or 3) (2.i) include inactive websites within their labeled datasets [311]. We note papers that perform website infrastructure analysis on contemporaneous snapshots (circa 2019) of websites in their corpora, even though the text-based features for the same analysis were drawn from datasets published in 2016 and 2017 [302]. This constitutes a serious source of (2.ii) temporal leakage. No works within this scope discuss approaches to accounting for uneven distributions in training data, or how they might account for shifts in baseline distributions during the lifecycle of an active website. Hounsel et al., for instance, train their classifier on a reference list in which 34% of misinformation training set sites were active and all websites in the real news training set were active, possibly resulting in overfitting to features specific to those inactive websites [1].

Model selection. Four of the seven methods we reviewed within this scope employed (3.ii) SVM classifiers; in two of those cases, SVM outperformed other, more complex unsupervised models [287, 302]. These results accord with our earlier observation, in Section 4.2, that SVM classifiers are comparatively fairly performant on two-way classification tasks.

Feature selection. Works within this scope employ multi-modal feature sets comprising a mix of (4.i) textual, (4.ii) network-based, and (4.iv) infrastructural signals: for instance, Baly et al. [316] consider network traffic, URL semantics, and site contents; and Hounsel et al. [1] consider TLS/SSL certificates, web hosting configurations, and domain registrations. None of the detection methods we reviewed discusses the computational costs of deploying their methods at scale. Though all works discuss their feature selection process (via leave-one-out and use-one-only evaluations), none describes a process for normalizing or weighting features according to dataset distribution or detection setting needs.

Evaluation. Methods within this scope that propose to perform whole-site labeling from analysis of a selection of news articles or infrastructural features are susceptible to distributional imbalances (e.g., between news verticals represented in an article corpus). Baly et al. train their model on (5.ii) political news websites only, and their credibility labels are strongly correlated with political bias scores. Hounsel et al. perform (5.i) testing in real time—one of the few studies we reviewed, and one of two studies at this scope, that did so [1, 109]. Both of these works report significant performance dropoffs between experimental and real-time tests, most likely as a result of distributional differences between real-world and experimental datasets (in practice, most websites do not host news-related content at all) [1, 109].

5 REPLICATION STUDIES

We choose three distinct targets and scopes to replicate issues identified in our literature review. Our targets are 1) (A.i) syntax-based text features; 2) (U.ii) user behaviors; and 3) (W.iv) multimodal whole-website features, including hosted content and infrastructure. These works are highly representative of their respective information scopes: Nasir et al. [141] (Section 5.1) train a neural network on corpora of true and misinformative news articles; Saeed et al. [361] (Section 5.2) use published lists of trolls to infer the presence of other troll accounts on Reddit; and Baly et al. [43] (Section 5.3) examine a multimodal feature set comprising infrastructure, content, and network-based features in order to infer whole-site credibility. For each work, we evaluate replicability and generalizability (RQ3, RQ4). Where possible, we 1) replicate reported results; 2) inspect datasets for potential dependencies; 3) perform ablation analyses to understand individual feature performance; and 4) test on current data.

Paper selection criteria and author outreach. We sorted our full text corpus by information scope. Within each scope, we sorted papers in order of decreasing citation count. We then proceeded as follows:

1. We attempted to source the full methods and datasets for the most cited paper within each information scope.

2. If we were unable to find this information during an independent web search, we reached out to the paper’s lead author(s) to request access.

3. If this request was unsuccessful—if the author did not respond, or confirmed that the dataset or code was no longer available—we returned to step 1 for the next most highly-cited paper in our corpus within that scope.

Replication analyses. In order to reproduce results published in a selection of papers and perform cross-cutting analyses on out-of-domain and out-of-sample datasets, we conduct a series of replication analyses on a subset of papers. We chose representative methods from disparate information scopes, and which consider a variety of different feature types. We reproduced results reported in each paper on the datasets mentioned therein, contacting authors when necessary to obtain datasets and code. We evaluated reproducibility and generalizability as follows:

• Reproducibility. We reproduce published results with code and data reported in the original publication. We evaluate availability of code and data and, where possible, compare our analysis outcomes with those reported in the original paper (RQ3).

• Explainability. Toward understanding the contributions of specific feature types to overall classifier performance, and why certain approaches work, we perform feature ablation studies when appropriate.

• Replicability and generalizability. Model perfor-mance on novel datasets is useful for determining the generalizability of existing detection methods to different contexts (RQ4). For models that were explicitly tested on specific misinformation narratives (e.g., 2020 stolen election narratives), on specific timeframes, or on specific types of misinformation (e.g., parody, satire), we develop updated datasets to test method performance on diverse information domains.

Ethical Considerations. Datasets used for this work were already publicly available or were obtained with permission from study authors. Our outreach study was approved by our institutional review board.

5.1 Detection of suspicious language

In view of rampant data dependency issues identified in our survey of article-scoped literature, we reproduced results from a representative study published in 2021 by Nasir et al [141]. The authors propose a neural net-based approach to the classification of news articles. The method employs a hybrid deep learning model that combines convolutional and recurrent neural networks for the classification of real and fake news. The authors report results for tests on two datasets: the ISOT dataset (45,000 news stories, equally distributed across true and false categories, as labeled by PolitiFact) and the FA-KES dataset (804 news stories about the Syrian war, 426 true and 376 false) [312, 313]. We were able to replicate original paper results and run the method on updated datasets of news articles. Motivated by the prevalence of methods trained on few- or single-source datasets in the article scope, we test possible dependencies related to journalistic house style, as all true articles in the Nasir et al. training corpus were sourced from Reuters.

House style as a confounder. To investigate house style as a possible confounder for misinformation detection, we excerpted 100 articles from both Reuters and The New York Times, two news outlets with distinctive (and different) reporting styles. We randomly selected these articles from both outlets’ RSS feeds in May 2023. We sourced articles for both corpora from the following verticals: U.S. and world politics, economics, science, and entertainment. Excerpt lengths ranged from 100 to 300 words. These corpora, each comprising 100 true news stories, are labeled “Reuters-original” and “NYTimes-original” in Table II.

Dataset (size)	Features	Acc.	FPR	FNR
ISOT (45,000)	original	0.995	0.00	0.00
	scrubbed	0.987	0.01	0.00
FA-KES (804)	original	0.521	0.151	0.843
Reuters (100)	original	0.727	0.71	–
	modified	0.507	0.02	1
NYTimes (100)	original	0.673	0.705	–
	modified	0.492	0.735	0.286
ChatGPT (250)	original	0.939	0.02	–

TABLE II: Replication analysis of Nasir et al. (2021). We tested the method of Nasir et al. on both datasets discussed in the original paper, and on novel datasets from Reuters and The New York Times [141].

We then selectively edited 50% of the news articles within each corpus. We changed proper nouns, negated verbs, and altered reported statistics so that the factual content of these articles was no longer accurate but house style and tone were preserved. We call these altered text corpora “Reuters-modified” and “NYTimes-modified” in Table II. The classifier had 0.727 accuracy on Reuters-original and 0.673 accuracy on NYTimes-original; this difference is not significant (p > 0.05). Additionally, the classifier had about 0.50 (random) accuracy on both modified datasets. The classifier’s false positive and false negative rates on modified and original Reuters and NYTimes corpora tell a more interesting story, however: Overwhelmingly, false NYTimes articles were classified as true (FPR = 0.02 and 0.735 for Reuters-modified and NYTimes-modified, respectively), while all true Reuters articles were classified as false (FNR = 1 and 0.286 for Reuters-modified and NYTimes-modified, respectively). These results indicate that the classifier was significantly more conservative in its assignment of true labels for the Reuters dataset than it was for the NYTimes dataset; additionally, within each modified corpus, the classifier did not effectively differentiate between true and untrue news articles. We did not investigate the stylistic features of each corpus that might have produced (anti-)conservative label distributions; for this work, however, it is sufficient to note that classifier performance in an adversarial setting was no better than random, and that house style appears to have a significant impact on classifier sensitivity to misinformative texts.

5.2 Detection of suspicious accounts

We reproduce results from a study published in 2022 by Saeed et al. [361]. In summary, the authors propose a method, called TrollMagnifier, for the identification of Reddit accounts that exhibit troll-like behaviors. Like all other account-scoped methods we analyze in the security literature, TrollMagnifier is trained on posting and reply statistics for non-troll and known Russian troll accounts (as identified by Reddit) [361].

Reproducibility of published results. Study authors provided us with pre-processed datasets and classifier code upon email request. The full Reddit Pushshift dataset is freely available online [421]. We were able to replicate original paper results using these materials. Original account handles were anonymized; as such, we were unable to manually verify if the accounts identified in the original study appeared to be troll-like.

Figure 1: Partial dependence plots for each TrollMagnifier feature. Respectively, x1 = “comments on posts that trolls commented on,” x2 = “comments on posts that trolls started,” x3 = “direct comment in reply to troll post,” x4 = “threaded comment in reply to troll comments on a troll post,” x5 = “same title post as troll,” x6 = “same title post as troll.”

Tautological targets. As described in preceding sections (Section 4.3), suspicious account detection suffers from a lack of clear and consistent definitions. Troll accounts cannot be described by degree of automation (while some trolls are bot-like, many others are operated by humans) [367] or the nature of the information they spread (this might be political misinformation, ads, or anything in between) [422]; as such, the clearest definition that Saeed et al. implicitly offer is that a troll is an account that exhibits troll-like behavior: i.e., interacts with known troll accounts, or appears on the same posts or message threads as these accounts. The authors note in their own work that this approach cannot be used to detect novel trolling behaviors, and requires a seed list of known troll accounts for every new detection task (we note in our discussion of Account-scoped data curation practices that the process of identifying these seed accounts is actually a fairly manual one).

Implied versus actual data dimensionality. While the original TrollMagnifier paper strongly implied that the proposed method would leverage networked behaviors to identify troll accounts acting in coordination online, we found that, in actuality, the features under analysis lacked any sort of temporal component and were limited in scope. There were six features in all (described in full in Figure 2), each corresponding to an aggregate engagement statistic. Longitudinal data and timestamps were not available; as such, it was not possible to perform time series analysis. Additionally, account names were not available, which disallowed construction of user graphs.

Feature importance. In our partial dependence analysis, we find that feature x1—commenting statistics—was the sole feature that consistently produced classification accuracy greater than 0.6 (most other features had accuracy no better than random; two features, x2 and x6, had performance below 0.5). Per our earlier observation that many account-scoped methods target behaviors that are difficult to distinguish from routine online activity, we recommend that feature engineering for account- and network-scoped methods reflect some intuition about the nature of actually suspicious behaviors. Furthermore, the performance of the current feature set—no better than random—suggests that manual classification might be as effective as (or even more effective than) an automated approach that only detects a content-agnostic heuristic.

5.3 Detection of suspicious websites

We reproduce results from a study published in 2018 by Baly et al [43]. In summary: the authors propose a multimodal approach to the classification of news websites. This method is particularly representative of works within this information scope: 50% of works within this scope use a similar mixed-modalities approach to detecting misinformation websites, and Baly et al. include site-specific feature types, including domain and traffic-based features, in their analysis. Baly et al. analyzed website contents, associated social media accounts, and Wikipedia pages in order to perform two classification tasks: fact and political bias classification. The authors developed a dataset of 1066 websites manually labeled for their political leaning (extreme-left, left, left-center, center, right-center, right, and extreme-right) and degree of credibility (low, mixed, high). These labels were extracted from the Media Bias/Fact Check (MBFC) database [311]. We were able to reproduce original study results and perform ablation analyses on existing datasets. We were unable to run the method on an updated dataset, as feature extraction code was not available.

Reproducibility of published results. All features extracted for the original analysis were captured in a series of json files. While we were able to readily reproduce results reported in the paper, certain elements of the dataset (follower counts on social media, Wikipedia page contents) were out of date. As we did not find documentation in the method repository for re-extraction of these features, we were restricted to conducting our tests on data that were already available. We binned bias labels into left, center, and right categories, as the seven-way taxonomy initially applied to the dataset by MBFC yielded small label classes. Classifier performance on the resulting three-way bias classification task accords with the results reported by Baly et al. on the same task.

Multimodal features: help or hindrance? We performed an ablation study of the method on the EMNLP18 dataset and analyzed the method’s bias and fact-checking classification functions separately [43]. Specifically, we stratified the original EMNLP18 dataset by political leaning and credibility, as labeled by MBFC, and analyzed the performance of 1) the full feature set, 2) individual features and 3) ablated feature sets (removing one feature type per test). Our results are summarized in Section A. We find that, on 11 out of 12 test datasets, classifier performance using only text-based features (articles and wikipedia, derived from articles randomly sampled from the website in question, and the site’s corresponding Wikipedia page, respectively) was comparable to performance on the full feature set. On five out six datasets, bias classification accuracy on text-only features actually outperformed bias classification on the whole feature set (see the bottom half of Section A), suggesting that the full-site classifier of Baly et al. was effectively a text content classifier.

6 DISCUSSION

We focus our discussion of results on those issues identified by our literature review and investigated in greater depth in our replication analyses. Additionally, we provide recommendations for evaluating ML-driven trust and safety interventions and link these recommendations to major takeaways of the present study.

RQ1: Fit. Very few methods that claimed to detect misinformation performed actual fact verification: Instead, they targeted proxy signals that were frequently steps away from promised detection targets. These differences were particularly noticeable in methods that relied heavily on text- and network-based features to perform classification. In those cases, semantic/syntactic signatures and propagation patterns served as proxies for the existence of misinformation. We demonstrated, through our own replication studies, that it is easy to circumvent approaches that rely on style-based cues to perform proxy detection. Additionally, in the absence of current or complete social media post data, we argued that most network-based methods are poorly-equipped to perform early detection of false rumors—this is by design, as the propagation patterns these methods purport to detect are frequently only noticeable after virality.

RQ2: Data curation and model explainability. Lack of access to current, well-annotated datasets remains a serious problem for current and future misinformation research. Across existing datasets, taxonomies for classifying misinformation were inconsistent. Testing on contemporaneous data was uncommon among those papers we annotated, and testing in real-time settings was even rarer. Proof-of-concept experiments oftentimes did not control for data dependencies. Authors describing black-box methods—particularly those employing neural nets or other forms of unsupervised learning—did not disclose feature sets retrieved by their methods. Model explainability for these approaches—if available at all—was very limited. For methods employing multimodal feature sets, temporal consistency across disparate feature sets was poorly controlled, and feature normalization was often questionable.

RQ3: Reproducibility. We noted widespread code and data availability issues: Fewer than 30% of our attempts to locate code and data, or obtain this information from authors, were successful. The code and datasets we were able to retrieve were frequently unusable or out of date. In fact, we were able to reproduce and replicate results on published and current data for only one of our replication studies. In that single case, we found that the article-scoped method performed no better than random (0.50 accuracy) on a current, mixed-domain dataset (Section 5.1). Additionally, for the significant number of methods that purported to detect false rumors on Twitter, the shutdown of the Twitter Academic API poses an existential risk to reproducibility of results and usefulness of dehydrated datasets.

RQ4: Generalizability. Methods that were trained on single-source or single-domain datasets appeared to perform well on data from the same source, or within the same domain; these methods, unsurprisingly, performed poorly on multi-source or out-of-domain topics. These discrepancies are closely tied to undisclosed or uncontrolled-for data dependencies, which we discuss in RQ2. Though we did not attempt to test detection methods on out-of-scope data—e.g., we did not test claim-scoped methods on datasets of whole news articles—it seems unlikely that these methods would fare well in out-of-scope contexts, particularly in light of their already-poor performance on in-scope datasets.

6.1 Future Directions for Research

Following our replication analyses and survey of the academic literature, we offer the following suggestions for possible future directions for research.

What works? Though the majority of this project has been devoted to critique of an existing body of work, we choose to highlight here a number of approaches that we found especially promising. In general, we found methods that 1) clearly described their detection targets and heuristics and 2) provided context for how their detection outcomes might be employed in service of fact-checking work the most methodologically sound [108, 17, 130]. As we note in our survey of commercial fact-checking services, the preponderance of high-volume checking work is hybrid, with (e.g.) potentially problematic content surfaced by automated means and ultimately vetted by human moderators. We recommend that future research efforts be devoted to understanding how best to facilitate collaboration between human and machine moderation techniques—in particular, how automated methods can support human moderators.

Definitive, rather than descriptive, targets. We discuss the knock-on effects of a basic lack of agreement about what does/n’t constitute misinformation throughout the main body of this work: In the absence of a well-defined detection target, detection methods detect proxy signals instead. Though the task of defining misinformation should likely not be left solely to computer scientists, we can nonetheless set our sights on more manageable targets, such as the three below.

Nuanced detection tasks. We note that, in general, the detection methods we consider in this work elide subtleties that are particular to the medium under consideration: for instance, article-level detection methods generally apply broad ‘true’/‘false’ classifications to a text under analysis where only single sentences or turns of phrase might be slightly inaccurate. Additionally, the language signals that most methods expressly detect are fairly unsubtle: Suggestion, insinuation, and leading questions are powerful rhetorical tools that might render a newsreader more susceptible to actual misinformation, or that might suggest misinformative ideas via indirect means; no works within our literature corpus expressly targeted these forms of language, however.

Multimedia detection. The detection methods we consider for this work are solely text-based; we note, however, that a good deal of social media misinformation is transmitted via audio, image, and video. This problem has become especially salient in the current year, as election misinformation transmitted via audiovisual means is rampant on messaging apps worldwide.

Designing for evasion resistance. The AI/NLP literature we review aligns with the current dogma in those subfields, which emphasizes generalizability of single models to all detection settings. We argue that a security framework for method development—wherein attempts at evasion and attacks are assumed, and iterative development is required to stay a step ahead of adversaries—is a more practical approach to misinformation detection research, and will yield more durable outcomes in the long run.

Availability

Our online SI is available in the following anonymous repo: https://anonymous.4open.science/r/sok_misinformation-B297/).

7 App: Full taxonomy of targets and critiques

[enumerate] Targets [ ] Claims [i.] Content-based detection via distance calculations on semantic embeddings [ii.] Content-based detection via search on knowledge graph topology [iii.] “Checkability” or “checkworthiness” [ ] Articles [i.] Syntactic and stylometric signals, including genre and sentiment [ii.] Topic-aware detection of stance and relevance to known rumoring topics [ ] Users [i.] Account metadata (bios, images, account age) [ii.] Single account behaviors (comments on posts, published posts) [ ] Networks [i.] Propagation patterns across social graphs [ii.] Timestamped records of user-user interactions [ ] Websites [i.] Text and URL/domain semantics [ii.] Site visitor demographics [iii.] Suspicious UI elements [iv.] Hosting infrastructure (DNS certificate, site age) Datasets [i.] Dataset age (is it current or out of date?) [ii.] Leakage (evidence of temporal leakage, feature leakage) [iii.] Data dependencies (are they accounted/controlled for?) [iv.] Availability of information required to reproduce or reconstruct similar datasets (was non-public information required to produce ground-truth training sets?) [v.] Availability of original data Models [i] Distance calculations on semantic embeddings [ii.] “Traditional” ML (SVM, RF, DT) [iii.] Deep learning (CNN, LSTM, GRU) [iv.] Graph cut algorithms [v] Stacked ensemble classifiers [vi.] Graph clustering algorithms Features [i.] Textual [ii.] Network-based [iii.] Author-, user- or source-based [iv.] Infrastructural Evaluation [i.] Testing in real time [ii.] Generalizability of approach [iii.] Evasion-resistance [iv.] Distributional shifts in training or test data [v.] Implications of high false-positive/false-negative rates

A Annotation Guide

Our guide for annotating papers follows below:

Target selection:

• What is the stated target of detection?

• What is the actual target of detection?

• Is the detection method directly performing fact-checking and misinformation detection, or is it detecting proxy signals instead (e.g., emotional language, topics associated with rumoring narratives)?

Dataset curation:

• Note dataset size, source, and temporality.

• Note information scope: what is the “operative unit” of checkable information? Where this might become more complicated: some multimodal methods, particularly network-based ones, usually accept some combination of user- and post-scoped data, but only make a final decision about the veracity of one of those things.

• Note information domain: does the data set coalesce around a topic / a set of topics? Are these pre-defined by the study authors?

• Fine to name multiple scopes if one info scope is a vital parameter / input to the model in its determination of the veracity of another; e.g., a model that performs network analysis of social media posts that share a link to a specific news article; the model might make a final determination about the truthfulness of the news article, but propagation patterns + social media posts were a crucial element of that decision.

Feature set selection:

• How were features selected?

• Are features explainable and accessible? Were they handcrafted? Were they generated by unsupervised models?

• Do positive signals indicate evidence of actually suspicious behavior?

• Do authors account for feature noise?

Test-train split:

• Note how/if this split was observed, and the percentage of the total data set assigned to test and training sets.

• Note any overlap between both data sets (ideally, none).

• Note any evidence of temporal leakage (future data used to make a prediction about past events).

Cross-validation performed:

• Note size and number of folds.

• Note presence of any development data sets (what percentage of total data set was this?).

• Note presence of any hyperparameter tuning during cross-validation process – are these parameters published in-text?

Out-of-sample testing:

• Note presence, if any.

• What data sets were used for this?

• Note any evidence of leakage during this process.

Third-party references:

• Particularly for claim-based studies that require some form of ground-truth reference for fact-checking, what is the ground truth reference employed by the study?

• Is this ground truth site reliable and up-to-date?

• What kind of taxonomy does the ground-truth reference employ for labeling true / false statements?

• What baked-in biases or slants might exist in the ground-truth reference’s data set (e.g., reference only labels political news sites; ground-truth reference is run by a known conservative organization)?

Model choice:

Evaluation:

• Note dataset(s) used for evaluating classifier performance.

• How does the evaluation dataset(s) compare to testing and training datasets in terms of size, sourcing, contents, and temporality?

• How do authors account for distribution shifts in rumoring topics or activity types? Is there a road map for iterative updates to their method?

Dataset (size)	All features	articles
	traffic
	twitter
	wikipedia
	url
	$-$         $+$
Full corpus (1066)	0.654	0.631	0.644	0.654	0.508	0.648	0.550	0.627	0.606	0.638	0.533
Med. corpus (400)	0.623	0.608	0.630	0.620	0.488	0.635	0.500	0.590	0.588	0.623	0.495
Small corpus (250)	0.636	0.632	0.596	0.632	0.524	0.608	0.512	0.588	0.536	0.624	0.516
Left bias (398)	0.691	0.683	0.671	0.688	0.668	0.686	0.628	0.678	0.683	0.678	0.636
Center (263)	0.913	0.810	0.890	0.913	0.700	0.924	0.741	0.920	0.776	0.890	0.635
Right bias (405)	0.279	0.267	0.252	0.279	0.173	0.286	0.230	0.274	0.205	0.272	0.121
Full corpus (1066)	0.569	0.523	0.595	0.569	0.399	0.580	0.440	0.552	0.538	0.577	0.373
Med. corpus (400)	0.563	0.517	0.580	0.560	0.420	0.578	0.478	0.585	0.545	0.568	0.360
Small corpus (250)	0.456	0.424	0.560	0.452	0.364	0.500	0.408	0.400	0.496	0.444	0.436
Low cred. (256)	0.590	0.516	0.633	0.590	0.641	0.629	0.445	0.609	0.633	0.590	0.473
Mixed cred. (268)	0.407	0.340	0.474	0.407	0.0522	0.414	0.258	0.362	0.276	0.414	0.198
High cred. (542)	0.349	0.336	0.408	0.349	0.255	0.369	0.271	0.341	0.316	0.351	0.218

TABLE III: Replication analysis of Baly et al.: Dropout(–) and feature importance(+) analyses of subsets of Baly et al.’s EMNLP18 dataset, stratified by political leaning and credibility. Most (secondmost) performant feature, as determined by its contribution to overall classifier accuracy on the full feature set, is highlighted in darker (lighter) hues. Fact and bias classification task performances are reported in the top and bottom halves of the table, respectively.

B Commercial fact-checking services

We include here a brief market survey of commercial and LLM-powered fact-checking and IO detection services. In general, these services fall into five categories: 1) media fact-checking organizations; 2) brand safety and suitability services; 3) trust & safety operations at large social media platforms, 4) threat detection operations, and 5) analytics organizations unaffiliated with a media outlet that offer research capacity to governments and businesses. We define each service category and (with the exception of the first category, which comprises human media workers and fact-checkers) discuss automated content moderation operations deployed by three prominent exemplars within each service category. In general, in instances where such information is made available, we observe that at-scale content moderation businesses at least employ human-labeled datasets to train classifiers, and some retain subject-area experts to adjudicate complex moderation decisions. On social media platforms, in particular, human moderators and automated systems appear to work hand-in-hand: automated systems surface potentially misinformative content that receives final verification from a human moderator. For IO detection, specialized knowledge (pertaining to specific geographies, languages, or political climates) is often invoked.

Brand safety and suitability companies. B2B companies that detect categories of potentially harmful speech on websites where ads might appear. Advertisers wishing to protect “brand safety” contract with these services to ensure that their ads do not appear alongside problematic content. The Global Alliance for Responsible Media (GARM) is the standards-setting body for brand safety and suitability companies [403].

• Zefr, a GARM member company, deploys AI to detect material that falls within predefined subcategories of problematic content (e.g., explicit content, misinformation, spam). In a press release for Zefr’s acquisition of an AI-driven content moderation company (AdVerif.ai) from 2022, the company disclosed that AdVerif.ai is “powered by fact-checking data from more than 50 IFCN-certified organizations around the globe” [404]—that is, AdVerif.ai trains its models on labeled datasets produced by (human) IFCN affiliates.

• DoubleVerify, a GARM member company, “uses sophisticated approaches that rely on a combination of AI and comprehensive human review” [405]. According to the company’s documentation, human assessors (a “semantic science team”) evaluate site infrastructure and contents; AI is used to scale their assessments.

• Integral Ad Science (IAS), a GARM member company, deploys AI to detect low-quality sites via infrastructure features. The company’s data sources, and deployment methodology were not immediately evident upon web search; IAS recently announced a new partnership with Meta for ad placement management on Facebook [406].

Trust & safety operations. In-house content moderation teams at large social media platforms.

• Facebook partners with IFCN affiliates to perform third-party manual checking of possibly misinformative content; first-line automated methods detect potentially harmful speech and surface near-duplicates of known problematic image (SimSearchNet++) and text content [352, 339, 407].

• Twitter has deployed a crowd-sourced annotations platform called Community Notes (formerly Birdwatch) since 2021 [408].

• TikTok employs thousands of content moderators across the globe who “work alongside automated moderation systems” [409, 410].

Threat intelligence services. At-scale detection of advanced persistent threats, foreign influence operations, and other cyberattacks oftentimes perpetrated by nation state actors.

• Mandiant strongly implies the use of hybrid detection methods, and disclaims that “defenders must constantly explore different techniques and leverage both subject matter expertise and technical capabilities to filter and uncover malicious activity”) [411].

• Microsoft Threat Intelligence strongly implies the use of hybrid detection methods; in a report from September 2023, MTI cites the work of in-house “Microsoft Security teams” which are tracking an advanced social engineering attack [412]. Other details—including possible use of automated methods—are undisclosed.

• Facebook Coordinated Inauthentic Behavior reports share quarterly updates about Meta’s takedown of coordinated activities across its platforms and others, including local news outlets. In a report from February 2023, Meta describes a CIB network in Serbia that used local news media to create the impression of grassroots support for the Serbian Progressive Party; while the nature of the detection methodology is unspecified, the complexity and geographic specificity of the CIB described suggest that specialists with country-level expertise were likely consulted [413].

Analytics firms. For- and non-profit organizations that offer checking services and research capacity to governments and businesses.

• The Global Disinformation Index (GDI) “reviews news domains based on various metadata and computational signals.” Content, however, is manually reviewed by a “country expert,” who analyzes a random sample of 10 articles from a news site to determine veracity [414].

• DFRLabs (Digital Forensic Research Lab) has disclosed that it employs human subject-area experts, and primarily addresses technology and policy issues pertaining to global and international affairs. In 2018, Facebook contracted its services to detect online trolls [415].

• Graphika Labs leverages network analysis to identify influence operations online. On its own website and in the popular press, Graphika has disclosed that it uses AI to map online networks and trace information flows [416, 417].

LLM-driven detection. A few LLM-powered detection methods have been discussed in the popular press, including those advertised by Google [418] and OpenAI [355], but these deployments appear to be mostly experimental, or have required additional adjudication from human moderators. OpenAI in particular has advertised content moderation tools that address misinformation-adjacent tasks, such as toxic speech detection [419]. Misinformation and toxic speech detection are not equivalent tasks, however, and the latter is narrowly defined in the Perspective training data documentation as a four-way classification task (the four class labels are “profanity/obscenity,” “identity-based negativity,” “insults,” and “threatening” language).