Understanding stance classification of BERT models: an attention-based framework

Abstract

BERT produces state-of-the-art solutions for many natural language processing tasks at the cost of interpretability. As works discuss the value of BERT’s attention weights to this purpose, we contribute to the field by examining this issue in the context of stance classification. We propose an interpretability framework to identify the most influential words for correctly predicting stances using BERT models. Unlike related work, we develop a broader level of interpretability focused on the overall model behaviour, aggregating tokens’ attentions into words’ attention weights that can be semantically related to the domain and proposing metrics to measure words relevance in correct predictions. We developed a broad experimental setting to analyse the premises underlying our framework regarding word attention scores and the capability concerning interpretability, adopting three case studies of stances expressed on Twitter on issues about the pandemic, and four pre-trained BERT models. We concluded that our method is not affected by the characteristics of BERT-models vocabularies, that words with high absolute attention have a higher probability of positive influence on correct classification, and that the influential words represent the domains. We observed many common words compared to a baseline method, but the words yielded by our method were considered more relevant according to a qualitative assessment.

Case studies: stances on issues about the COVID pandemic

We consider three case studies related to the pandemic to explore attention weights as the basis for interpreting stances in BERT classification models. Our research group developed two of them as we assessed the influence of political polarization for stances expressed in Twitter about social isolation [25] and vaccination [26]. These studies include a topic modelling analysis using BERTopic [34] to understand the main arguments that support each stance. The third case study addresses stances on hydroxychloroquine as a treatment for COVID-19 [36]. Table 4 summarizes the main characteristics of each case study relevant to our purposes.

(a) Social isolation: This case study reflects Brazilians’ polarized position at the beginning of the pandemic by late March 2020. President Bolsonaro launched a campaign “Brazil cannot stop" as a reaction to the social isolation measures, claiming that the damages to the economy were more extensive than the health benefits. Supporters of this campaign are referred to as “Chloroquiners". In contrast, the opponents (“Quarentineers") focus on the need to protect lives. We identified a strong political bias praising or criticizing the President and the central government. Further details can be found in [25, 33].

(b) Vaccination: This case study reflects Brazilians’ polarized position regarding COVID-19 vaccination from January 2020 to April 2021. Pro-vaxxers praise the science and demonstrate joy/anxiety/relief regarding vaccination, while anti-vaxxers regard vaccination as an individual choice. There has been a heated discussion about mandatory vaccination to reach collective immunity, with a Supreme Federal Court (STF) ruling about the constitutionality of this measure. A strong political bias was also identified, including a specific anti-vaxxer stance, referred to as “anti-sinovaxxers”. This subset represents the political dispute between President Bolsonaro and Sao Paulo’s governor João Dória, using as a target the so-called Chinese vaccine Coronavac, a partnership between a Brazilian and a Chinese institution. Coronavac was the first vaccine available to Brazilians, and it was politically exploited by Doria, who, back then, was a prospective candidate for the 2022 Presidential election. Details are provided in [26, 27].

(c) Hydroxychloroquine: This case study focuses on the polarization among Twitter users about using chloroquine and hydroxychloroquine to treat COVID-19. The data were collected during April 2020 and cover various events related to these drugs, such as India’s hydroxychloroquine export ban, the publication of clinical trials results, the accusations against the White House about a political/economic interest behind the push for these drugs and the FDA warning about the use of these drugs. There are three stances: people in favour of the use of these drugs (Pro-chloroquine), people against (Anti-chloroquine) and people without open opposition/acceptance of this treatment (Neutrals). More details on this dataset can be found in  [36]. Since the original analysis is limited to the most frequent words in the dataset, we deployed BERTopic for topic modelling for further understanding.

Table 4 Case studies, stances and main arguments

In all three studies, stances were assigned according to the presence of specific pre-defined hashtags, which were removed from the datasets to avoid bias for stance classification. We deployed typical pre-processing actions such as removing mentions/URLs/special characters, lower casing and discarding short tweets (less than three words). Details can be found in the original studies. We selected random samples from the original datasets for our experiments, as detailed in Sect. 5.