Viblio: Introducing Credibility Signals and Citations to Video-Sharing Platforms

YouTube is the leading video-sharing platform with 2.6 billion users worldwide [45, 61]. While YouTube serves as an important source of information, with one in four US adults turning to the platform for their news [49], it is also a fertile ground for misinformation and conspiracy theories about health, vaccines, politics, climate change, social issues, and more [26, 60, 64, 65]. To compound the issue, the platform’s recommendation algorithms have been criticized for driving people down “rabbit holes” of misinformative content [57, 60]. In the past, YouTube has tried to tackle the problem by removing misinformative videos from the platform, pushing mainstream channels as top search results, suppressing the recommendations of problematic information, and demonetizing misinformation-spreading channels [92, 102, 103]. YouTube also introduced informational interventions, such as the inclusion of Wikipedia links within videos promoting conspiracy theories [41] and the integration of fact-checks from independent organizations into search results [73]. Some of these interventions have demonstrated efficacy, reducing traffic to anti-vaccine videos on the platform [67]. However, despite these efforts, YouTube continues to grapple with misinformation. In this research, we aim to address this issue by exploring methods to increase user awareness of underlying inaccuracies in videos. Our approach focuses on combating misinformation in the case that it does reach viewers by allowing them to explore intentional credibility-related signals and practice lateral reading techniques.

In the digital era, the internet has brought forth an abundance of information sources, offering people an extensive array of options for finding information. Faced with this deluge of information, people often exert little cognitive effort in deeply reflecting on the validity of online information and instead may rely on heuristics to make assessments about the content [30, 84, 105]. Such heuristics are often triggered by online signals or cues such as structural characteristics (e.g., length of content), communicator characteristics (e.g., likability, appearance), or audience characteristics (e.g., audience’s reactions) [30]. Scholars have posited that there are various cues embedded in the affordances (modality, agency, interactivity, navigability) provided by online platforms that can activate cognitive heuristics shaping users’ perception about the credibility of online content [101]. When a group of users collectively like or share a news article on social media, these actions signal collective trust in the information’s credibility [48, 56]. Additionally, agency affordance can also activate the authority heuristic when users perceive the source of information as authoritative. This occurs because individuals are inclined to place more trust in content originating from experts, reputable organizations, or authoritative accounts [22, 72].

While online platforms present numerous signals across all four aforementioned affordances, not all of these signals lead to sound judgments. Research has demonstrated that people may form inaccurate judgments based on easily manipulable signals, such as the number of sources cited in an article [99] or visual aesthetics [75], or during the sensemaking process [32, 69, 70]. Therefore, it is important to identify and understand the role of diverse signals present on online platforms that influence users’ perceptions of online credibility. On top of the signals present on online platforms, recent work using Community Notes (formerly Birdwatch) has found that contextual features, such as the partisanship of users, can affect a user’s judgment of whether other users’ tweets are misleading [10]. While prior research has explored credibility signals on platforms such as X (formerly Twitter) [42, 72, 76, 76] and Facebook [8, 17], as well as on news websites [85, 100], and the broader web ecosystem (see [35] for a review), there exists a significant gap in research concerning video-based platforms like YouTube. While previous work has studied the features of video content that affect users’ engagement with videos [80], we still lack an understanding of how users evaluate the credibility of content on this type of platform. Our exploratory study aims to bridge this gap by identifying the diverse signals on YouTube that users rely on to evaluate and trust video content, while also highlighting the challenges users face with existing credibility signals.

To combat online misinformation, both scholars and online platforms have employed a wide array of strategies and approaches [7]. These include establishing policies against misinformation for platform governance [54, 92, 98], reducing the visibility of misleading content by suppressing and down-ranking it from searches and recommendations [3, 44, 89], as well as identifying and removing misinformative content [4, 12, 86]. Platforms have also embraced measures like de-platforming and demonetization to limit the influence of repeat spreaders of misleading content [29, 31, 53]. Concurrently, efforts to raise public awareness about misinformation have been amplified through targeted media literacy programs [23, 63, 93]. Furthermore, scholars have also explored several design interventions to aid individuals in assessing the credibility of online content [15, 33, 40, 68, 90, 97, 106]. They include interventions targeting online content, including message-based approaches conveying the detrimental consequences of misinformation [34], incorporation of warning messages [33, 68], inoculation messages [39], and debunking of false information through fact-checks and corrective messages [20, 24, 25, 51]. Recognizing that the source of information plays a pivotal role in shaping individuals’ credibility assessments, several design interventions seek to redirect attention towards source information and the authoritativeness of the source itself [15, 40, 90, 97]. Furthermore, scholars have explored the potential of crowdsourced judgments in evaluating the credibility of online information [15, 18, 59]. Crowd-based interventions have also been successfully adopted by online platforms. For instance, X launched the Community Notes initiative that allows users to identify and annotate tweets that they believe contain misinformation or false content [38].

In our work, we built a prototype system called Viblio which leverages crowdsourcing to incorporate credibility citations directly into YouTube videos, providing users with additional contextual information while they are viewing video content. Along with results from our exploratory study, we took inspiration from Community Notes and Wikipedia to inform the design of Viblio. Community Notes provided an example of how crowdsourcing could directly influence both viewer’s trust in a source as well as their active engagement in incorporating credibility awareness into their usual viewing habits. Wikipedia showed us how citations can impact trust [78] and allowed us to envision how citations could be embedded within a video as well as what type of videos could benefit from citations.

Participants were primarily recruited through word-of-mouth and X. Participants were asked about basic demographic information like age range, gender, and education level, and were then screened for comfort with screen sharing and being recorded in online meetings (Table 1). Generally, participants were selected by sign-up order and to diversify the participant group as much as possible. Of the 12 participants (Table 1), the majority were in the 18-22 age range and currently in an undergraduate program (7 out of 12). Overall, the participants skewed female (8 out of 12) and were under the age of 30 (10 out of 12). Recruitment materials presented the study as one focused on misinformation and credibility on YouTube. Each participant was compensated $15 for their time.

With IRB approval from our institution (STUDY00013111), interviews were conducted virtually over Zoom across a 3–week period. Each interview lasted approximately 60 minutes and was recorded and transcribed using Zoom’s recording software. Before conducting each interview, the participant was asked for consent to record the session and told that they could stop the interview at any time if needed. All searches were done in an incognito browser window on the participant’s computer to prevent the participant’s YouTube or Google recommendations from influencing the study. This study was conducted prior to YouTube’s removal of dislike counts on videos. The format of the interviews borrows from contextual inquiry methods and a semi-structured interview format. Participants were given 4 tasks to complete, in order: (1) video selection using results page signals, (2) comparing YouTube and Google results, (3) video credibility using playback page signals, and (4) citation generation while watching videos. Once each of the 4 tasks were completed, we asked participants open-ended questions about their process and choices. At the end of each interview, we asked participants general questions related to their YouTube usage and their knowledge of citations and credibility. The activities participants were asked to complete are as follows:

The three interviewers used open coding methods to perform a preliminary round of independent qualitative coding using the interview transcripts. After discussion, a consolidated code–book was established and used by the primary interviewer to code the transcripts. Based on the code–book, emerging topics were identified by thematic analysis. Codes included but were not limited to result order on the search results page, video length, video channel verification, video thumbnail and preview, date posted, title, production quality, description box, comments, suggested videos, production quality, and resemblance to clickbait (sensationalized title and/or thumbnail).

In this study, we found several signals in which participants commonly relied upon (Table 2). The most popular included familiarity with the YouTube channel, production quality, and the video description. It was also revealed, however, that the participants were skeptical of the effectiveness and reliability of certain signals. Some signals, such as YouTube’s verification system and popup banners, were misinterpreted by some participants. Participants also suggested potential improvements and new approaches to credibility on YouTube, including knowledge panels on search results pages and introducing citations.

Prior to implementing our system, we developed some design goals for Viblio based on the findings of our exploratory study, other credibility-focused systems, and extant literature. For functionality, the system should allow anyone to add a citation and view all citations on any YouTube video. We wanted to add granularity to these citations by only displaying them for a certain time frame, as perhaps only a part of the video requires a citation. Feedback from interviewees regarding their usage of citations showed that it was important to specify the purpose of a citation, such as whether it supports or refutes content in the video or simply provides further information. Participants also expressed that they would only want or need citations in specific circumstances, so we wanted the system to be low-profile when not in use.

As we designed Viblio, we kept our findings from the exploratory study and prior work in mind. Community Notes’s initiative that allows users to identify and annotate tweets that they believe contain misinformation or false content inspired our use of a crowdsourcing approach for the collection of citations [38]. Based on extant literature ([101]), we wanted to activate users’ agency in shaping their perception of content credibility. It was important for us to be able to display information from experts, reputable organizations, or authoritative accounts, as users are inclined to trust this content more [22, 72].

Our exploratory study shows that existing credibility signals can play a large role in users’ credibility judgment. We thought it was particularly interesting how users went to look at the video description on the playback page, which influenced our decision to place the extension below the video. It was also surprising how easily signals could get misinterpreted, like YouTube’s information panels on misinformation-prone topics. As a result, when designing our system, we wanted to make it clear that the extension was a separate add-on to the video that referenced citations and not any sort of endorsement by outside sources. Although we purposefully kept the citations separate, we also did not want the interface to distract or interfere with the user’s experience watching videos. As P10 mentioned in 3.2.2, we wanted our system to be able to blend into the interface when not needed. For this reason, the elements, colors, and shapes within the extension closely mirror YouTube’s existing interface.

Another important factor we considered in the design process was indicating the citation type. It was important for users to be able to quickly identify whether a citation supported or refuted the information in the video. Based on suggestions from participants, we also chose to include a third category of citations, extra information. This type is intended for sources that provide further context or expand upon the information provided within the video. We also wanted to be careful about how this labeling would affect the viewer’s overall opinion, especially as recent work found that stance-based labels may intensify selective exposure and lead to users being more vulnerable to polarised opinions and fake news [50].

When considering the permission-levels and potential for misuse of Viblio, we wanted to focus first on whether citations made in good faith would have an impact or have any utility for users and how best to present citations. After validating this, future work could consider how to guard against bad-faith actions. We discuss possibilities in Section 7 (Future Work) that draw ideas from successfully moderated crowd-powered systems.

Citation insertion. When the user wants to add a citation, they can click the “Add citation” button on the extension. This opens up a form where users can input citation information such as the link to the citation, the type of citation, any comments or notes, and start/end times for the section of the video it references (Fig. 3a). The start time is automatically populated to the current video playback time, and the end time is 10 seconds later than the start time (or the end of the video, if that is sooner). Both fields can be adjusted by the user. On the backend, information about the source is scraped from the provided link, such as the title, author, source, description, and cover image.

Timeline view. The default view on Viblio is the timeline view (Fig. 1). A timeline is displayed across the top of the Viblio window, which syncs with the video timeline. If a user jumps ahead in the video, the Viblio timeline will mirror this. Any crowdsourced citations added to the video will appear as circles along the timeline. As the YouTube video plays, once the video enters a time frame specified by a citation creator, the relevant citation will be displayed under Viblio’s timeline. Users can also navigate to the start time of the video clip associated with a citation by clicking on the corresponding circle on Viblio’s timeline.

List view. The other way to explore citations is through the list view (Fig. 3b). By default, all of the video citations are displayed in a scrollable list. Shortcut buttons to individual citations are shown on the left side of the window. The citations on this page are not displayed based on when their specified time frame is currently being played, so they give the user a chance to explore all of the citations in depth.

Citation types. From our contextual inquiry, we learned that users may use citations for a variety of purposes, such as references or a bibliography, flags for misinformation, and to provide extra information. As a result, we wanted to incorporate a way to represent these different types of citations in the system. We included options to “support the video clip claim”, “refute the video clip claim”, and “provide further explanation” (as seen in Fig. 3a). Depending on the type of citation, an icon is displayed on the citation, and a color treatment (green for supportive, red for refuting, or blue for informative) is used throughout the extension in reference to the specific citation, such as in the citation’s coordinating circle on the timeline. Using colors and icons to indicate the citation type allows users to quickly identify the relationship of the citation to the video.

Citation format. When a citation is displayed within Viblio, a consistent format is used. We based the information collected and displayed for each citation on popular citation formats and the norms set forth by Wikipedia. Depending on the information that was able to be scraped from the provided link, the citation block will display the title, source, description, and cover photo. Along with the metadata, any comments or notes the user made when adding the citation, the timespan the citation is referenced in the video, and the type of citation is displayed.

This extension was implemented for Chrome using JavaScript (React). The back end is an Azure CosmosDB database that interfaces with a React front end through a Python Flask API deployed as an Azure Web App. We chose to develop a Chrome extension for YouTube over a separate video-playback platform as our goal was to build upon users’ existing behaviors and an established social platform.

However, this decision also required some trade-offs due to the limitations of developing a Chrome extension. The main trade-off we made was that Viblio would only be able to be used on a computer and would not be able to be used on the YouTube app or mobile Chrome. A more integrated approach would allow Viblio to be used on the app or mobile devices, responsible for around 90% of YouTube’s watch time [6]. In the meantime, our prototype allows us to quickly experiment with and deploy new credibility signals in real-world settings.

Participants were primarily recruited through word-of-mouth and X and were asked to provide demographic information, including age range, gender, political preference, and education level (Table 3). They were also asked about their familiarity with YouTube, the type of content they typically watched, and the average amount of time they spent on YouTube (hours/week). Responses to recruiting materials were then screened for comfort with screen sharing, being recorded in online meetings, comfort using Chrome, and basic knowledge using YouTube. Generally, participants were selected by sign-up order with an aim to also diversify the participant group as much as possible. Of the 12 participants, half were in the 23-30 age range, and three were over 45. Overall, the participants skewed male (7/12), educated (7 with Master’s, 5 with Bachelor’s), and usually spent up to 7 hours on YouTube a week (6 spent less than 3, 5 spent 3 to 7 hours). The group was relatively politically diverse, with 3 identifying as Democrats or left-leaning, 2 identifying as Moderate or centrist, 1 identifying as Republican or right-leaning, 1 identifying as independent, and 4 choosing not to share. Upon completion of the entire study, participants were compensated with a $60 gift card. One participant (P6) did not fully complete the study; however, his responses were included for the exercises he did complete.

The study took place virtually on Zoom over a period of around 4 weeks, although each participant took a maximum of two weeks individually to complete the study. Each participant made their way through three distinct sections of the study: an onboarding interview, independent activities using Viblio, and a reflective interview.

On-boarding interview. For the on-boarding interview, we started by assisting the participants through downloading and installing Viblio. Depending on the participant’s technology experience, this sometimes involved screen-sharing to provide a visual aid. Once Viblio was installed, participants were given the link to an onboarding–specific YouTube video. Participants were then given time to explore and interact with Viblio. Once they felt comfortable with the extension, we then led participants through usability exercises. These exercises involved navigating to the Timeline and List views, viewing specific citations, and adding a new citation to the video. For the purpose of the usability test, the content of the citation added was not important. Once the onboarding and usability exercises were complete, any remaining time was spent ensuring that participants fully understood how to use Viblio before the independent exercises.

Independent exercises. The main portion of this study is the independent exercises each participant completed. Over the course of around 15 days, each participant was given a schedule of videos to watch, split across ten days. Each day, participants had 2–3 videos to watch (totaling under 30 minutes) and a reflective survey to complete. Participants were divided into three randomized groups, and each group received a unique schedule. The schedules were planned so that each group would encounter videos at different stages. Some videos would not have any citations added, while others would already be populated with citations by other participants. This method ensured that participants encountered a variety of citation volumes across videos.

Daily survey. For each day of the study, each participant would fill out a daily survey for the videos they watched. Participants were asked to fill out this survey while completing the exercises so that responses would be of the moment. Before watching each video and interacting with Viblio, the participant would rate how credible they believed the video was on a scale of 1–5 and explain their reasoning behind the rating. Once the participants watched the video and interacted with Viblio, they would then rate the credibility again. If their score changed, they were asked to explain what had affected their rating (or not). If the participant came across something in the video that they believed needed a citation and there was not one already added by another participant, we asked them to find an appropriate source and add a citation. Participants would also record this event in the daily survey.

Video set. We developed the set of videos used in the study to cover a variety of topics, types of video, possible source types, use cases, and credibility levels (Table 4). The topics covered by the videos went from informative topics like the history of cereal and laser eye surgery, to controversial subject matters like election fraud, COVID-19 vaccines, and abortion. The types of video included educational, entertainment-based, and news reporting. Videos were published from well-known sources like Fox News and Crash Course, as well as from small local news stations and independent creators. A few of the videos purposefully contained problematic or false information. During the reflective interview, participants were informed of the videos, including misinformation.

Reflective interview. Once each participant finished the independent exercises, a semi–structured final reflective interview was scheduled. The first part of the interview focused on the participant’s experience throughout the study. Questions were asked about the usability, their process for exploring and adding citations, how they chose whether or not to add a citation, how Viblio affected their YouTube experience, how Viblio affected their credibility ratings, and the quality of citations they came across. The participant’s responses to the daily survey were reviewed beforehand, then any outliers or noteworthy responses were discussed. This discussion helped clarify motivations, the participant’s process, and any confusion. The last topic of the interview covered possible applications of the system. Participants were asked to evaluate if they could see themselves using this system in the future and, if so, where they would use it.

Analysis. The three sections of the study were analyzed individually. Open-coding methods were used to complete a sample set from each section. Once a code-book was established, the three data sets, the transcripts from the onboarding and reflective interviews and the daily survey, were coded. Based on the code book, emerging topics were identified by thematic analysis. Some of the codes included but were not limited to design feedback, usability, use case, citation quality, and process. Throughout the study, data was also recorded whenever users interacted with Viblio and the citations. Given the small sample size, the statistical results of this data collection are not significant to our findings. We do, however, refer to this data to discuss participant’s usage of Viblio throughout the study. Usage data results can be viewed in Tables 5 and 6 in the Appendix.

One of the key takeaways from our study is the extent to which Viblio empowers users to form their own opinions about video content. Participants emphasized that Viblio’s value extended beyond the mere presence of citations; rather, it provided them with a crucial evaluative point alongside existing signals in assessing credibility. Despite the value users mentioned, it was apparent that Viblio worked alongside other pre-existing signals rather than replacing them. While participants acknowledged that citations themselves could harbor biases, they still considered them invaluable for fact-checking and validation. Intriguingly, some participants proposed the incorporation of a user-rating metric for individual citations, reflecting an eagerness to refine the evaluation process further. As noted in Section 2, a group of users collectively liking or sharing a news article on social media signals collective trust in the information’s credibility [48, 56]. Provided this finding proves true in Viblio’s application, the additional user-rating metric could provide another signal that users could employ in their credibility-determination process. This revelation underscores Viblio’s potential as a catalyst for greater user engagement and discernment in the face of misinformation.

Despite Viblio’s potential as a tool against misinformation, it is still susceptible to issues other crowd-based or social platforms face. In the current version of Viblio, anyone, including bad actors, can add citations. In some cases, this could lead to a video, full of misinformation, citing a multitude of untrustworthy and/or incorrect citations. To combat this, it is critical to safeguard against low-quality and malicious citations. While this study focused solely on implementing an initial system for trust and credibility, it is imperative to integrate moderation tools in the future. Similar studies and projects, like X’s Community Notes, can be used as an example to develop Viblio further.

One possible approach that Viblio can follow is that of Wikipedia’s crowd-sourced citations, which set standards contributors must follow [5]. Another system to look to is Community Notes, which has already implemented both a reputation system for users and a voting system for individual ’notes’. So far, Community Notes has been shown to increase the quality of content produced and decrease the number of people sharing content [19]. Preliminary findings from Community Notes suggest a similar success for Viblio if implemented at scale. It is important to note the differences that Viblio faces in comparison to similar tools on other platforms. While Community Notes focuses on small amounts of text-based data, Viblio must be able to address multiple viewpoints and information points for each video. One tweet may only touch on one point, but a YouTube video has the possibility to address a range of opinions and facts. Viblio has the ability to provide viewers with a way to make an overarching opinion on the credibility of the video and to determine the validity of individual statements. Viblio also faces challenges when compared to Community Notes or Wikipedia in terms of the video format; citations for text have long-established norms and formats.

Another challenge area to consider for Viblio is in situations where the facts are not settled yet, or information is contested. In these situations, citations claiming to support or refute a fact may only confuse the sense-making process of users. This struggle has already become apparent on Community Notes [46]. One way Community Notes is attempting to combat this is by showing the community multiple notes suggested by users as well as upvotes to allow for the community to collectively decide what to eventually share. Given our participant’s use of like/dislike counters as a credibility signal in the explorative study already, a possible approach for Viblio is to include individual counters for each citation on a video.

A question that remains in regard to Viblio is the possible scalability of the application. The crowd-sourced approach that Viblio currently implements requires a high degree of user involvement in comparison to systems like YouTube’s information panels. However, unlike information panels, Viblio is able to provide customized approaches to each individual video that can mitigate issues found with information panels (see Section 3.2.1). Viblio, like Community Notes, could prove its strength for high-traffic videos, where more users are likely to add citations throughout a video. On the other hand, Viblio’s current system falls short for low-traffic videos which will be unlikely to have many (or any) citations added.

This shortcoming could be mitigated through the possibility of automating citations. While AI-generated citations would help combat low-traffic misinformation, the citations themselves may prove to be another possible source of misleading sources or misinformation. An option that warrants further exploration is a human-AI collaborative approach, where AI suggests possible citations that users approve or deny. Another future possibility AI can add is the match of current citations to other matches or near matches of claims/video sequences across different videos, allowing for a citation to be copied over.

An unexpected but enlightening finding from our study pertains to the potential mental load Viblio introduces into the user experience. Some participants noted that using Viblio alongside video content required a noticeable additional cognitive effort compared to their customary YouTube viewing habits. However, the unanimous consensus among all participants was that this added cognitive load was justified, particularly in the context of complex and controversial topics [62, 83]. This observation highlights Viblio’s role as a tool for enhancing critical thinking and information literacy, potentially fostering more thoughtful engagement with digital content.

Beyond its primary goal of aiding credibility determination, our study revealed that Viblio can serve a broader set of purposes, particularly in the realm of education. Participants envisioned Viblio as a valuable supplement to video content, allowing viewers to access in-depth sources and expand their knowledge on specific topics. For instance, in educational contexts such as computer science history, where videos often provide concise overviews, Viblio’s ability to offer quick and easy access to comprehensive sources can significantly enhance the learning experience. As educational content continues to proliferate on platforms like YouTube, Viblio stands as a promising tool with the potential to elevate the quality and depth of learning outcomes.

Our findings also shed light on various design choices and enhancements that could further refine Viblio’s utility. Participants’ feedback suggests the importance of exploring different placements for citations, adding filter views to facilitate the review of citations by type, and implementing a minimum show-time for citations, even if the cited period is shorter. Moreover, the idea of crowdsourcing the usefulness of citations, as proposed by one participant, holds promise in augmenting Viblio’s functionality, especially given preliminary findings for Community Notes [19]. Users’ varying responses to citations based on their initial credibility ratings emphasize the need for Viblio to cater to different user needs and perceptions.