Who Checks the Checkers?
Exploring Source Credibility in Twitter’s Community Notes

Abstract.

In recent years, the proliferation of misinformation on social media platforms has become a significant concern. Initially designed for sharing information and fostering social connections, platforms like Twitter (now rebranded as X) have also unfortunately become conduits for spreading misinformation. To mitigate this, these platforms have implemented various mechanisms, including the recent suggestion to use crowd-sourced non-expert fact-checkers to enhance the scalability and efficiency of content vetting. An example of this is the introduction of Community Notes on Twitter.

While previous research has extensively explored various aspects of Twitter tweets, such as information diffusion, sentiment analytics and opinion summarization, there has been a limited focus on the specific feature of Twitter Community Notes, despite its potential role in crowd-sourced fact-checking. Prior research on Twitter Community Notes has involved empirical analysis of the feature’s dataset and comparative studies that also include other methods like expert fact-checking. Distinguishing itself from prior works, our study covers a multi-faceted analysis of sources and audience perception within Community Notes. We find that the majority of cited sources are news outlets that are left-leaning and are of high factuality, pointing to a potential bias in the platform’s community fact-checking. Left biased and low factuality sources validate tweets more, while Center sources are used more often to refute tweet content. Additionally, source factuality significantly influences public agreement and helpfulness of the notes, highlighting the effectiveness of the Community Notes Ranking algorithm. These findings showcase the impact and biases inherent in community-based fact-checking initiatives.

1. Introduction

Misinformation on social media platforms has become a pressing issue which has captured the attention of Governments, policymakers, researchers and the general public (Rainie and Anderson, 2017). Several studies indicate that false information spreads faster than verified facts. This is particularly true for Twitter where information irrespective of its truthfulness can spread across a huge fraction of the population in a small duration of time (Vosoughi et al., 2018; Chakraborty et al., 2017). This rapid dissemination of false information poses a serious concern as it has tangible and impactful real-world consequences, such as, on public health, elections, and national security. For example, one of the most well-known instances that illustrates the severity of the issue in recent times was a series of tweets from former U.S. President Donald Trump where he falsely claimed that the 2020 US presidential elections were faked (Yen et al., 2020). This misinformation played a significant role in inciting a violent attack on the U.S. Capitol on January 6, 2021 (BBC, 2021).

Misinformation detection is the foremost step in mitigation of misinformation. Traditional methods often involve expert fact-checkers or specialized organizations that use their expertise to validate or debunk claims made on digital platforms (Moreno-Gil et al., 2022). However, this approach faces scalability issues as the volume of online content far exceeds the capacity of such experts to scrutinize it (López-Marcos and Vicente-Fernández, 2021). An alternative strategy involves the use of automated misinformation detectors (Sharma and Sharma, 2021), such as, machine learning techniques (Guo et al., 2022) which analyze the underlying linguistic patterns to distinguish between truthful and misleading content (Butt et al., 2022), (Sharma et al., 2022). Despite their utility, these automated systems are not infallible and frequently necessitate human moderation for optimal performance (Horne, 2023) and further, provide suggestions to combat misinformation (Sharma et al., 2023b), (Sharma et al., 2023a).

To tackle both of these challenges, social media platforms have moved more towards reliance on the wisdom of crowds for fact-checking (Allen et al., 2021). In 2021, Twitter introduced a community-based fact-checking service originally known as Birdwatch, which was subsequently rebranded as Community Notes (Biron, 2022). While the service was initially built to add useful context to tweets, it has recently taken a shift towards combating misinformation on the platform (Duffy, 2022). Community Notes allows users to flag misleading tweets and provide annotations that include context or corrections. Users have the option to further substantiate their notes by adding links to external sources. Experts on misinformation believe that partisanship is the main reason for spreading misinformation (Altay et al., 2023). Due to this, it is crucial for social meia platforms to minimize bias, especially in the context of fact-checking.

The aim of this research is to shed light on the community-driven aspect of fact-checking within Twitter’s Community Notes feature. With the increase in prevalence of misinformation, particularly in politically charged arenas, understanding the dynamics of community-based fact-checking is more critical than ever. While a few works have looked at community note user interactions and user consensus (Pröllochs, 2022; Pilarski et al., 2023), none have yet explored the sources used in Community Notes (See Figure 1). This research benefits platform developers, policymakers, and researchers by providing actionable insights into the sources and potential biases that may skew public discourse. We systematically evaluate the types of sources cited, analyze their biases, and factuality levels, and probe the impact of these variables on audience perceptions. We summarize the contributions of the study through the following research questions (RQs):

RQ1 - Sources of Validity: How do the citation patterns within Community Notes reflect and influence the biases and factual integrity of the information that shapes public opinion and fact-checking efforts? With this research question, we examine the patterns of web page citations within Community Notes to uncover how information sources influence public opinion and fact-checking efforts. Initially, we identify the most frequently cited web pages and investigate whether citation frequencies differ based on the country of origin of these pages, aiming to understand geographical biases in source popularity. Subsequently, we evaluate the bias and factuality levels of these commonly cited web pages to assess their impact on shaping collective viewpoints. This evaluation is crucial for identifying potential misinformation and understanding the overall reliability of the information disseminated. Furthermore, we explore correlations between the type of source, its bias, and its factuality, seeking to reveal systematic tendencies in the selection and use of sources. By examining these aspects, we aim to highlight areas where critical evaluation of sources is needed to enhance the credibility and factual grounding of shared information.

RQ2 - Perceptions of audience: How do source characteristics such as type, bias, and factuality impact their effectiveness in refuting or supporting Community Notes and influence the perceived helpfulness and agreement with them? In this research question, we investigate the roles that different categories of sources play in supporting or refuting content shared on social media platforms, specifically Twitter. Furthermore, we examine how these same attributes, source type, bias, and factuality, affect the perceived helpfulness of explanatory notes appended to tweets. This exploration seeks to determine whether notes rated as helpful are also those that are unbiased and fact-based. Additionally, we study the impact of source characteristics on the perceived agreement with the notes, assessing whether source credibility influences audience perceptions. Through this comprehensive analysis, we aim to highlight the complex interplay between source credibility and the reception of crowd-based fact-checking on Twitter.

These questions expand our understanding of how community-based fact-checking functions and provide critical insights for platform developers, policymakers, and researchers aiming to improve the efficacy and fairness of online information verification systems. The rest of the paper is structured as follows: In Section 2, we present an overview of the related works in the domain. In Section 3, we present the details about the data used for the experiments. In Section 4, we introduce the details of the experiments carried out and discuss the re- sults obtained. The conclusions and Ethics Statement are presented in Sections 5 and Ethics statement respectively. We additionally release the code used for the analysis of our results.

Figure 1. Screenshot of a tweet and its associated Community Note. The note itself cites sources used to fact-check the claim.

2. Related work

This section covers past works that deal with exploring bias in fact-checking and sources of information on social media, which are the focus areas of this study.

Fact-checking on social media: Fact-checking on social media primarily falls into three categories: expert, automated, and community-based (Bozarth, 2022). These branches have evolved to meet the unique challenges posed by the rapid dissemination of information on social media platforms(Chakraborty et al., 2019; Chakraborty and Chakraborty, 2023). Expert fact-checking platforms, such as PolitiFact (Li and Chang, 2023) and Snopes (Hannak et al., 2014) provide a thorough understanding of news articles and statements as they rely on professional fact-checkers. However, expert fact-checking requires extensive time and effort, which makes it difficult to handle the huge volume of information disseminated online (López-Marcos and Vicente-Fernández, 2021). Recently, automated fact-checking-based approaches have become popular, which leverage machine learning and natural language processing techniques for instant verification (Guo et al., 2022). Although these methods are highly scalable, they fail to provide justification and interpret the contextual information, which is essential for reliable and trustworthy fact-checking (Santos, 2023), (Nikopensius et al., 2023). To overcome these challenges, several existing research studies have proposed community-based fact-checking as a possible alternative. Community-based fact-checking is not dependent on a few expert individuals as it leverages the wisdom, allowing for several factual interpretations (Godel et al., 2021). However, while it solves the issues of fact-checking speed and explanability, it can still be susceptible to biases and manipulation (Saeed et al., 2022).

Bias in Fact-Checking: Bias can significantly influence human perception and the creation of fact-checks, particularly on issues that evoke strong negative opinions (Park et al., 2021). Understanding this is crucial, as these inherent biases affect users’ interpretation of information. Although several works indicate strong bias in fact-checking users for social media platforms, such as Twitter (Shin and Thorson, 2017), this has been explored only on a few expert-based fact-checking platforms, such as PolitiFact (Draws et al., 2022). We indicate that no research has studied and analyzed the bias in Twitter community notes, which is one of the reasons this study focuses on the issue.

Media source bias and factuality: Research on media sources is extensive and diverse, with a focus on various aspects that contribute to media bias. One avenue has been to investigate quoting patterns to discern biases (Niculae et al., 2015). Another line of research has examined how media bias is evident in the citations to think tanks and policy groups (Groseclose and Milyo, 2005). More recently, the influence of bias in headlines has also been studied (Pan et al., 2023). These traditional media studies lay the groundwork for understanding bias, but the landscape is continuously evolving with the growth of social media platforms. On Twitter, scholars have taken different approaches to analyzing media bias. One such method has been to align co-subscribers of news sources to deduce potential biases (An et al., 2021). Another approach has looked at the nature of reactions in Twitter comments to different news articles, offering insights into public perception and inherent biases (Spinde et al., 2023). Beyond user behaviour, it’s also crucial to consider the role of algorithms; for instance, studies have explored how Twitter’s algorithm amplifies content with varying degrees of bias (Huszár et al., 2022). Given the intricate interplay of user behaviour and algorithmic influence in shaping and amplifying bias, our study aims to delve deeper into the dynamics of community-based fact-checking.

Twitter Community Notes: Although there are a plethora of research works on Twitter datasets covering topics such as information diffusion (Vosoughi et al., 2018), sentiment analysis (Wang et al., 2022), content sources (Singh et al., 2020), etc., there are very few existing research works on Twitter Community Notes (Wojcik et al., 2022). Existing works on Twitter Community Notes includes that of Pröllochs et al. (Pröllochs, 2022), in which they empirically analyze the Twitter Community Notes by examining user interactions, note credibility, sentiment, and the influence of tweet authors on user consensus. Subsequent research has expanded to include comparative studies that examine Twitter Community Notes with respect to fact-checking, such as snoping and expert reviews (Pilarski et al., 2023; Drolsbach and Pröllochs, 2023; Saeed et al., 2022). However, none of these approaches performs a study of the bias and fact-checking of the sources. Therefore, in this paper, we focus on a multi-faceted exploration of sources and audience perception within Twitter Community Notes. Unlike prior studies that focused on the diffusion, consensus, and sentiment of users in reaction to Community Notes compared to expert fact-checks, our research aims to understand how source characteristics influence audience perceptions, thereby providing a more comprehensive understanding of source credibility’s role in shaping public opinion. We observe that mediabiasfactcheck.com, allsides.com, adfontes.com have been highly effective and reliable in the detection of source bias in both Twitter (Samory et al., 2020; Huszár et al., 2022) and Reddit (Weld et al., 2021; Bayiz et al., 2024a). Therefore, we utilize these media bias ranking platforms, to identify, understand and interpret bias in Twitter Community Notes which will inherently provide a more nuanced understanding of online crowd-based fact-checking.

3. Data

In this section we explain the data used for the study. We additionally highlight the key variables we use to refer to certain features of our data. We cover data related ethical questions under Section Ethics statement.

3.1. Community Notes

Figure 2. Types of top 50 sources (after combining related domains). The relative frequencies over the whole dataset are plotted on top of each URL bar.

Community Notes (previously called Birdwatch) is a community-driven fact-checking feature on Twitter which was launched in January $2021$ (Community Notes, 2024). The fact-checks on Community Notes are called ”notes” in short. To write notes, Twitter users must separately sign up for the Community Notes feature. Once verified, Community Note users can start reviewing tweets, writing notes and rating other Community Note users’ notes. For the Twitter end-user, only the highest-rated Community Note per tweet is displayed (the Community Note must also have at least $5$ ratings to be displayed). In addition, Twitter end-users can rate the top displayed note of a tweet. The minimum rating threshold restricts the spreading of bot-generated notes.

Data Collection: We downloaded all of the publicly available (royalty-free) Community Note data from the Twitter Community Note website (Twitter, 2023) from the period starting from January $23$, $2021$ and ending with January $27$, $2024$. The dataset is separated into four subsets. The first subset, Notes, contains information about all notes. The second subset, Ratings, contains information about the ratings of a note. The third subset, Note status history, contains metadata about notes, including what statuses they received and when. The final and fourth subset, User status, contains metadata about each user’s enrollment state. For our analysis, we used the Notes, Ratings and Note status history sub-datasets. The Notes and Note status history datasets comprise of $544995$ Community Notes from $87294$ users, and the Ratings dataset comprises of $6514542$ Ratings.

3.2. Key variables

We introduce the key variables we extracted using data mining from the dataset or additionally annotated. Content refers to the textual content of the note. This includes any supporting source links added by the Community Notes user as well. Source refers to the hostname of the URL that the author of the note refers to in the corresponding note as the source. Type refers to the type of the source. Bias refers to the bias rating of the source. Factuality refers to the factuality rating of the source.

4. Analysis

4.1. Sources of validity (RQ1)

In this subsection, we employ data collection, cleaning, and categorization techniques to understand the variety of sources cited in Community Notes. Our objective is to identify which publications and their origins are most commonly utilized and assess their contribution to the platform’s information veracity through analysing $Bias$ and $Factuality$.

4.1.1. Cited sources:

We, initially, investigate which sources are cited more (compared to others) and analyze the types of these sources. For this, we collect all the web links (44523 unique links in total) from the Content of the notes and then, perform the following pre processing. We initially simplify the URLs to their hostnames, such as reducing https://www.example.com/article/123 to example.com, address issues with short links, redirects, automatically retrieve the original sources from web archive links by using a script and accept redirect links only when originals were unavailable. The complete list of expanded URLs can be seen in Table 1.

Category	URLs
Link shorteners	tinyurl.com, www.shorturl.at,
	bit.ly, is.gd
Social media redirects	g.co, t.co, yahoo.com, goo.gl,
	youtu.be, redd.it, fb.me
Web archives	web.archive.org, archive.ph,
	archive.is, archive.org

Table 1. List of URLs Expanded

Table 2. Most Frequent Sources by Type in Top 500 Sources, including their contribution percentages in their corresponding type. Note that there were only two sources for the Web Archive category as most of them were already expanded.

Type	Top Sources	Percentage of Type
News	BBC, Reuters, AP	6.29%, 5.34%, 3.81%
Fact-checking	Snopes, Politifact, FactCheck	31.72%, 21.42%, 12.13%
Dictionary/Encyclopedia	Wikipedia, Britannica, Merriam-Webster	87.78%, 4.11%, 2.36%
Government/Civil	MHLW, WHO, Gov.uk	16.96%, 7.02%, 4.92%
Social Media/Platforms	Twitter, X, YouTube	48.54%, 18.60%, 14.82%
Research	NIH, CDC, USGS	15.35%, 12.07%, 8.08%
Web Archive	Wayback Machine, DOI	50.61%, 49.39%
Search Engine	Google, Justia, Bible Gateway	92.83%, 3.13%, 2.20%
Other	all-senmonka.jp, ne.jp, apple.com	15.39%, 12.77%, 8.69%

Furthermore, after the initial preprocessing, we group similar URLs that represent the same domain. This grouping includes different device versions of websites, such as en.wikipedia.org and en.m.wikipedia.org, and both short and long forms of websites like youtube.com and youtu.be. Additionally, we consolidate different country versions of websites, exemplified by bbc.com and bbc.co.uk. We also group subpages of the same institutional websites, such as twitter.com and help.twitter.com, recognizing them as originating from the same core domain. This approach helps our analysis by reducing redundancy and focusing on fundamental source identities. We consider the top $500$ URL groups on the basis of highest occurence frequency in the Community Notes data. These groups comprise of total $4064$ URLs and have been used in $306578$ notes (approximately 56% out of all notes in that period). For the rest of the paper, we refer to these groups as top $500$ sources.

We consulted mediabiasfactcheck.comsupplemented by the sources ”About” sections to categorize the top $500$ sources into $8$ Type categories: News, Fact-Checking, Dictionary/Encyclopedia, Government/Civil, Social Media/Platforms, Research, Search Engine, Web Archive and Other. Other category comprises of URLs that do not fit into any of the other categories, for example, private business pages, portfolios, download links, etc. On evaluation of the distribution of source categories, we observe a long-tail pattern, where a small number of sources are extremely frequent while the majority are cited less often as can be seen on the percentage mentioned with respect to each URL in Figure 2. For example, most of the sources are used in less than 1% of Community Notes. We additionally noticed that the category of News has a substantial portion (almost $50\%$) of the URLs in this distribution.

To understand the sources that dominate each category, we summarize the three most frequently cited sources across each Type category. Our observations indicate that most of categories depend on a few dominating sources as seen in Table 2, while only a few categories have a high variance in the sources as is the case with News. Wikipedia’s dominance within the Dictionary/Encyclopedia category, accounting for 87.78% of the citations in this type, highlights its critical role as a primary reference source. Additionally, the Fact-Checking category shows a substantial concentration among the top three sources (Snopes, Politifact, and FactCheck), contributing to a significant portion of the category’s citations with 31.72%, 21.42%, and 12.13% respectively.

Irrespective of the category segregation, we additionally observe in Figure 2 that Twitter is a highly cited page, being used in notes $60168$ times ($9.3\%$ in the whole dataset). The reason is that users primarily do intra-domain fact-checking, i.e., cite other tweets in their fact-checking notes. We highlight that some of these cited tweets might have cited other sources themselves, but as we do not have access to the tweets cited or the tweets for which the community notes were written, we had to exclude Twitter tweets from our analysis. However, this does not have a significant impact on our analysis or conclusions since any links hidden in these cited tweets most likely follow the same distribution of source types that we got from our annotation stage. We also keep Twitter and X ungrouped as sources due to the ongoing discussion regarding whether the platform has changed its political leaning after Elon Musk took the company over in 2022 (Bump, 2023; Eco, 2023). The second-most cited source (after Twitter) is Wikipedia, a web encyclopedia, which in itself is a community-reviewed platform. We additionally notice that Government/Civil and Research sources in the top $50$ are primarily about health and nature, which might indicate that fact-checking these topics requires more expert knowledge.

We also categorized the top 500 sources by country which is shown in Figure 3. Most of the sources are from English-speaking countries, with Japan and Brazil being the most frequent from non-English countries. This is expected as Community Notes opened contributor access to users from these countries first as is highlighted in Table 3.

Country/Region	Date
US	Jan 23 2021
Canada	Dec 15 2022
UK, Ireland, Australia, New Zealand	Jan 20 2023
Brazil	Mar 3 2023
Japan	Mar 21 2023
Mexico, Spain, Portugal	Apr 7 2023
Argentina, Chile, Colombia	May 4 2023
Ecuador, Guatemala, Peru, Venezuela
Italy, Germany, Austria	Jun 14 2023
France, Luxembourg, Belgium,
Netherlands, Switzerland, Slovakia	Jul 20 2023
Bulgaria, Croatia, Cyprus	Jul 26 2023
Czechia, Denmark, Estonia, Finland,
Greece, Hungary, Iceland, Latvia,
Lithuania, Malta, Norway, Poland,
Romania, Slovenia, Sweden, Indonesia,
Malaysia, Philippines	Nov 16 2023
Singapore, Thailand, Papua New Guinea,
Brunei, Algeria, Bahrain, Egypt, Israel	Nov 22 2023
Jordan, Kuwait, Lebanon, Morocco,
Oman, Palestinian Territories, Qatar,
Tunisia, United Arab Emirates
Hong Kong, South Korea, Taiwan	Dec 7 2023

Table 3. Community Notes Release Dates by Country

4.1.2. Bias and factuality of sources:

To better comprehend the influence of cited sources on the political leaning and factuality of fact-checking on Twitter, we analyze the Bias and Factuality labels of these sources. In this section, we explain the annotation process of Bias and Factuality and give a visual overview of the Bias and Factuality distribution of the top sources (including country-wise distributions). We first annotate all of the $500$ sources with metrics for Bias. We do this by aggregating Bias labels for these sources from three media monitoring websites: mediabiasfactcheck.com, allsides.com, adfontes.com. These websites have also been widely used for the same purpose in several existing research works (Rao et al., 2021; Samory et al., 2020; Weld et al., 2021; Bayiz et al., 2024b; Xiao et al., 2023). The media monitoring platforms state that the Bias class is given based on the content of the pages, guest lists, and political leaning on certain topics. We aggregate these Bias classes using majority voting, meaning we took the dominant class over all three. If the three classes do not agree and thus no dominant class was found then we removed the source from our dataset. The labels from mediabiasfactcheck.com also had a Pro-Science label for Bias, which we considered neutral as is expected from scientific sources. Additionally, as only mediabiasfactcheck.com have classes for Extreme Left and Extreme Right, we consider those classes as Left and Right correspondingly to maintain consistency across all the media monitoring websites. In the end we are left with $5$ Bias classes - Left, Left-Center, Center, Right-Center and Right. The Bias labels of each media monitoring page and our finalized aggregated labels can be seen in Table 4. After Bias label annotation our dataset comprises of $183$ sources which covers $991$ URLs and used in community notes $206466$ times.

Figure 3. Source Type Relative URL Count by Country. The total URL count is marked at the end of each bar. INT refers to global international sources (or sources that did not fit under any single country) and EU refers to European Union sources.

Figure 4. Source Bias Relative URL Count by Country. The total URL count is marked at the end of each bar.

Figure 5. Source Factuality Relative URL Count by Country. The total URL count is marked at the end of each bar.

MBFC	AS	AF	Bias final
Left, Extreme Left	Left	Strong Left	Left
Left-Center	Lean Left	Skews Left	Left-Center
Center, Pro-Science	Center	Middle	Center
Right-Center	Lean Right	Skews Right	Right-Center
Right, Extreme Right	Right	Strong Right	Right

Table 4. Bias classification across media monitoring platforms (MBFC: mediabiasfactcheck.com, AS: allsides.com, AF: adfontesmedia.com). Each row aligns similar classes to a consolidated final bias label.

Type			Bias			Factuality
Type	Count	Percentage	Bias	Count	Percentage	Factuality	Count	Percentage
News	$543$	$54.8\%$	Left-Center	$448$	$45.3\%$	High Factuality	$399$	$40.3\%$
Research	$177$	$17.9\%$	Center	$422$	$42.6\%$	Mixed Factuality	$227$	$22.9\%$
Social Media/Platforms	$90$	$9.1\%$	Right-Center	$79$	$8.0\%$	Mostly Factual	$211$	$21.3\%$
Dictionary/Encyclopedia	$88$	$8.9\%$	Left	$25$	$2.5\%$	Very High Factuality	$150$	$15.2\%$
Search Engine	$67$	$6.7\%$	Right	$16$	$1.6\%$	Satire	$1$	$0.1\%$
Fact-checking	$12$	$1.2\%$				Low Factuality	$1$	$0.1\%$
Government/Civil	$12$	$1.2\%$
Other	$1$	$0.1\%$

Table 5. URL counts by Type, Bias, and Factuality

We study the distribution of the Bias labels by domain country origin as shown in Figure 4. It’s interesting to note that more polarized (Left and Right) sources were those from the USA, Great Britain, Canada, Australia and India. This is most likely because the media monitoring companies are US-based and thus also more critically evaluate English-speaking sources. Additionally, we highlight that USA-based sources are most frequent, covering 640 URLs after Bias annotation.

We annotate our sources additionally with Factuality labels. However, as, mediabiasfactcheck.com, allsides.com and adfontes.com sources do not use the same features to identify Factuality, we could not aggregate these three sources and consider only mediabiasfactcheck.com. We chose mediabiasfactcheck.com out of the three as it provided maximum coverage for the sources. For mediabiasfactcheck.com, the Factuality label is based on the frequency of fact-checks they have passed during the last five years. There are $6$ Factuality classes - Very High Factuality, High Factuality, Mostly Factual, Mixed Factuality, Low Factuality and Satire. However, mediabiasfactcheck.com does not provide a Factuality rating for all sources. Therefore, to maintain consistency in our dataset, we exclude those data points for which we do not have any Factuality label. After adding Factuality labels, our final dataset comprises of $182$ sources, covering $990$ URLs and used in $206007$ community notes. We additionally analyse the Factuality class across country origin of the sources as can be seen in Figure 5. We can see that Low Factuality sources are entirely from Great Britain, which also has the largest proportion of Very High Factuality. This trend of having varied sources in terms of Factuality is also seen in other English-speaking countries such as the USA and Australia.

The sources after adding Type, Bias and Factuality labels is considered our final sources dataset, which we use to analyse the sources used in Community Notes. Every source has one Type, Bias and Factuality annotated category and we show their URL count distribution between categories in Table 5. We analyze the interrelationship between categories to understand the patterns of information framing and its impact on public perception. For example, our observations as shown in Figure 6 show the connectedness of Type, Bias and Factuality categories. We highlight, that the majority of News outlets ($50.5\%$) lean towards a Left-Center bias, and of those, a substantial amount ($79.6\%$) are highly factual.

Figure 6. Connectedness of source categories of Type, Bias and Factuality. The nodes are weighed by URL counts.

4.1.3. Correlation analysis:

To get a better understanding of how different categories intersect and interact with each other, we analyse the correlation of the Type, Bias, and Factuality categories. We mark the Pearson correlation coefficients as $r$ and consider scores with $r<0.3$ weak, $0.3<r<0.7$ moderate and $r>0.7$ strong. We exclude scores where $r<0.3$ (weak correlation) from our analysis and display our results in Table 6.

	News	Research	Center	Right
Center	$-0.38$	$0.37$	-	-
Very High Factuality	$-0.5$	$0.5$	$0.46$	-
Mixed Factuality	-	-	-	$0.31$
Low Factuality	-	-	-	$0.35$

Table 6. Correlation of Type, Bias and Factuality categories. Shown are scores, with absolute values of r ¿ 0.3.

Our results reveal a few moderate correlations ($r>0.3$) that hold critical implications. Firstly, Research sources are often categorized as both Center ($r:0.37$) and having Very High Factuality ($r:0.5$). This suggests that such sources are both reliable and seen as advocating scientific perspectives in an unbiased way. Secondly, News sources are generally less likely to be categorized as Center ($r:-0.38$) or possess Very High Factuality ($r:-0.5$), which implies a potential limitation in these commonly-accessed information outlets. Thirdly, Center sources tend to also score very high in factuality ($r:0.46$), reinforcing the credibility of unbiased perspectives. Lastly, Right biased sources frequently exhibit Low Factuality ($r:0.35$) and Mixed Factuality ($r:0.31$), raising questions about the credibility of such sources and their role in public discourse.

4.1.4. Summary of insights for RQ1:

We emphasize key points about the sources cited in notes. First, Twitter and Wikipedia are the most cited, suggesting that intra-domain fact-checking and community-reviewed content are significant in public discourse. This shows that social media platforms are not just arenas for discussion but also crucial sources of information and fact-checking. The fact-checking source Type category showcases a significant reliance on a few key websites, with Snopes, Politifact, and FactCheck which forms a substantial portion (65.27%) of citations. The Government/Civil category, with top sources being MHLW, WHO, and Gov.uk, reflects the diversity and international representation of credible government and civil sources utilized for fact-checking. Secondly, the Bias and Factuality show most sources fall within Left-Center (54.8%) and High Factuality (40.3%), indicating a factual left-leaning fact-checking community. Thirdly, sources from English-speaking countries tend to be more frequently used and also more polarized in regards to bias and factuality. Lastly, News sources correlate with not being Center, Center and Research sources correlate positively with having Very High Factuality and Right biased sources with Low Factuality and Mixed Factuality.

4.2. Perceptions of audience (RQ2)

Figure 7. Distribution of community notes by Bias and Factuality, categorized as either supporting or refuting Tweets. Amount of notes representing each bar is added on top. For the second plot we have also added z-score 99% confidence intervals.

In this section we investigate how the Factuality and political Bias of cited sources influence the ratings and acceptance of a community note. Specifically, we look at which sources are used to support and refute notes, what is their helpfulness and how agreement levels differ in source usage. We also highlight how well the Community Note rating algorithm handles poor quality and biased content.

As one community note can have several sources associated we need a way to aggregate Bias and Factuality labels of sources used. For this we calculate Bias and Factuality scores for each community note. The associated scores that correspond to each level can be seen in Table 7. We consider Satire equal to Very Low Factuality score-wise as for fact-checking hidden humour can be misleading and more harmful than useful. To deal with multiple sources in community notes we disregard notes that have sources from opposite Bias sides and average the Bias scores otherwise. For Factuality, we average the Factuality scores.

Factuality to Score		Bias to Score
Factuality Level	Score	Bias Category	Score
Very High Factuality	5	Left	2
High Factuality	4	Left-Center	1
Mostly Factual	3	Center	0
Mixed Factuality	2	Right-Center	-1
Low Factuality	1	Right	-2
Very Low Factuality	0
Satire	0

Table 7. Factuality and Bias levels converted to scores.

We use a simplified 3 class system for both Factuality and Bias. The score to label transformation system is highlighted in Table 8. We highlight that the largest Factuality class is Medium (61.4% of notes) and the largest Bias class is Center (51.67% of notes).

Category	Score Range	Count	Percentage
Right	$<-0.5$	10,227	6.60%
Center	$-0.5$ to $0.5$	80,101	51.67%
Left	$>0.5$	64,688	41.73%
Low	$<3$	44,497	28.70%
Medium	$3$ to $4$	95,179	61.40%
High	$>4$	15,340	9.90%

Table 8. Distribution of Bias and Factuality scores with labels, counts, and percentages.

Figure 8. Distribution of community notes by Bias and Factuality, categorized as either helpful, not helpful or needs more ratings. Amount of notes representing each bar is added on top. For the second and third plot we have also added z-score 99% confidence intervals.

4.2.1. Role of sources in supporting or refuting the content of tweets:

Community notes can be used to support both the truthfulness of a tweet (marking the original tweet as not misleading) or refute it (marking the original tweet as misleading). This label is given by the community note writer. We aim to understand which categories of sources are used more for supporting and which ones are for refuting tweets. We can see the Bias and Factuality distributions of the refuting/supporting notes on Figure 7.

On Bias, we can see that notes with Right-wing sources are used relatively more when supporting tweets compared to notes that use Center or Left-wing sources ($p<0.01$). This could indicate a broader tendency within the conservative media space to create self-reinforcing loops of information (Jay D. Hmielowski and Beam, 2020).

Our observations on Factuality indicate that sources with lower factuality are used relatively more to support tweets than those with high factuality ($p<0.01$). This can be an indication of misinformation enforcement, where a community note has been written to a misleading note to make it seem credible.

4.2.2. Role of sources in community note helpfulness:

Community Notes undergo a contributor-driven rating process to determine their status as ”helpful”, not helpful”, or ”needs more ratings”, affecting their visibility on site timelines and posts (Notes, 2024). Initially, all notes start in a ”Needs More Ratings” state until receiving at least five ratings, at which point they may be classified as helpful or not helpful. Notes identified as fact-checking potentially misleading tweets that meet specific helpfulness score criteria are marked as helpful and displayed on posts, whereas those not meeting the criteria are deemed not helpful. The process includes a diligence scoring mechanism to evaluate the accuracy and sourcing of information, ensuring that notes recognized as reliable and clear by a broad spectrum of users are highlighted. We can see the Bias and Factuality distributions of the helpful and not helpful notes in Figure 8.

We aim to look at the Bias and Factuality of sources used in helpful and not helpful notes. When it comes to Bias, we notice that Left and Right sources are associated relatively more frequently with helpful than not helpful notes compared to Center sources ($p<0.01$). This might be due to people having a confirmation bias when looking at content confirming their views.

However, regarding Factuality we notice a trend where notes that hold low factuality sources are generally rated less helpful than those with medium or high factuality sources ($p<0.01$). This confirms that notes with lower quality sources are effectively classified by the Community Notes Ranking algorithm. The Factuality distribution of the helpful and not helpful notes is shown in Figure 8.

4.2.3. Perceived agreement by source categories:

We analyse the perceived agreement levels of the Community notes per source category. For this, we use the existing ratings associated with each note and created an agreement index using the number of ratings that agreed and disagreed with the note:

Figure 9. Distribution spread of aggregated agreement by factuality and bias categories. The black dotted line indicates average agreement.

(1) $$\text{{Agreement}}=\frac{{\text{{Agree}}}}{{\text{{Agree}}+\text{{Disagree}}}}$$

We indicate a threshold of $0.5$, where scores greater than $0.5$ indicate that the notes are more agreeable than disagreeable, while scores lower than $0.5$ suggest the opposite. We plot the distribution spread of aggregated agreement for our categories of Bias and Factuality in Figure 9. We also notice that the average agreement per our community notes is 0.87.

We explore whether the bias of the cited sources in notes correlates with the level of agreement those notes receive. We assume that notes citing more politically neutral sources will attract broader agreement. This is based on the assumption that neutral sources may be less likely to polarize opinion compared to clearly left- or right-leaning sources (Mitchell et al., 2014). On analyzing the Bias category results, our initial observations indicate that notes that cite Right sources have significantly lower agreement levels, with the lower quartile showing an agreement score under $0.5$. However, notes that cite Left sources show similarly high levels of agreement to the notes that use Center sources.These findings highlight that notes which cite more politically center or left leaning sources are generally more agreeable than those relying on Right sources.

Factuality plays a large role in the agreeableness of statements; thus, we expect that sources with higher Factuality ratings are also generally more agreeable. When we inspect the Factuality category agreement levels, we see that higher Factuality sources, also have generally higher aggreement levels and vice versa for lower Factuality sources. Notes that cite sources of Low factuality see the lowest agreement levels, with their lower quartiles being below the 0.5 level. This indicates that community note users pay attention to the factuality of the source when rating the notes.

4.2.4. Summary of insights for RQ2:

We summarize the main takeaways of this analysis next. Our results shows a significant preference for right-leaning sources to support the content of tweets which hints at a tendency among conservative media outlets to form self-reinforcing information cycles. Similarly, we found that community notes that support tweets often rely on sources with low factuality scores, therefore highlights a systemic issue with misinformation. Further, our study indicates that notes associated with either left or right sources tend to be considered helpful, whereas those that cite sources of lower factuality are not, demonstrating the Community Note rating algorithm’s ability to effectively filter content quality. Finally, we note that community notes citing more neutral or factually sound sources receive higher agreement levels, emphasizing the importance of source quality in achieving community agreement.

5. Conclusions

Our investigation into Twitter’s Community Notes has uncovered distinct patterns in the use of sources for community-led fact-checking, showing clear trends and biases. We’ve discovered that Twitter and Wikipedia are often the go-to sources, highlighting a preference for checking facts within the platform and relying on community-reviewed information for public discussions. Our findings reveal that sources are mainly left-center in bias and high in factuality, suggesting a left-leaning trend in the fact-checking community. Moreover, sources from English-speaking countries are used more frequently, indicating a bias towards these regions and a more pronounced polarization in terms of political bias and factual accuracy. This polarization is especially evident in the types of sources cited, with news sources often showing clear bias, whereas academic and research sources are typically linked to very high factual content. In contrast, right-biased sources are often associated with lower levels of factuality. Adding to this, our results indicate a noticeable preference for right-leaning sources to support tweets, which may suggest that right-wing users are creating echo chambers on the platform. We also observed that notes endorsing tweets often depend on less factual sources, pointing to a broader issue with misinformation. Interestingly, our analysis shows that notes linked to both left and right biases are usually seen as helpful, except when they reference lower-quality sources. The low agreement of low-quality sources justifies the usage of ratings of notes used by the Community Note rating algorithm in sifting through content quality. Additionally, notes that cite more balanced or factually accurate sources tend to receive higher levels of agreement, underscoring the critical role of source quality in fostering community consensus.

Ethics statement

We emphasize that we rely entirely on publicly available and anonymous data shared on Twitter Community Notes; hence, we do not and are unable to obtain consent from users who wrote notes to tweets. We add that this research did not involve any human subjects or crowd-workers, and all annotations were done by the authors of the work itself using reliable materials. We discuss the data handling in depth to mitigate any misuse of our results.