Assessing the Impact of Music Recommendation Diversity on Listeners: A Longitudinal Study

In its simplest form, Impact Assessment (IA) can be defined as “[...] the process of identifying the future consequences of a current or proposed action. The impact is the difference between what would happen with the action and what would happen without it" [39]. In this area, AIA is relatively a new field in comparison to other frameworks such as Environmental Impact Assessment (EIA) [63], Social(and Societal) Impact Assessment (SIA) [51, 92, 93], Human Rights Impact Assessment (HRIA) [47], or Cultural Impact Assessment (CIA) [70]. Nevertheless, its importance is quickly growing.

AIA can be defined as a set of “emerging governance practices for delineating accountability, rendering visible the harms caused by algorithmic systems, and ensuring practical steps are taken to ameliorate those harms” [62]. A few aspects of AIA are addressed in detail below, but we point the reader interested in a comprehensive overview and discussion on current AIA practices towards the reports published by the non-profit organisations AI Now Institute [76] and Data & Society [65].

As discussed by Vecchione et al. [94] in the context of algorithmic auditing, most of the impact that may result from the interaction with an algorithmic system appears beyond discrete moments of decision-making. This is particularly true for RS, wherein the impact may not be evident after a single interaction, but instead be the fruit of multiple exposures through time. Second, as Metcalf et al. [62] argue, the impact is co-constructed by all the actors linked to an algorithmic system: developers, designers, decision-makers, public and private organizations, and, most importantly, the affected communities. Therefore, it is fundamental to involve each of these actors while defining the AIA practice, a vision shared also in Reference [94].

While the exploration of the long-term impact is already on the agenda of RS practitioners, the involvement of the wider community of people affected by RS is still rare to find. In the music field, a notable exception is the work by Ferraro [21] considering the artists’ perspective on the impact of music recommendation. Another example that is worth mentioning is the work done in the Algorithmic Responsibility research area at Spotify [84].

As recently observed by Liang and Willemsen [56], longitudinal studies are not common in RS literature and their recent work is a notable exception, as also the study presented by Hauptmann et al. [36] presenting a user study on a nutrition assistance recommender system. On the contrary, the interest in impact-oriented RS research using synthetic data and simulation environments is rapidly growing within and outside the RS community [18]. However, this growing interest is accompanied by an equally growing concern about the high heterogeneity and low transparency of methods, evaluation practices, and more general assumptions on which such simulation studies are built [99]. Next, we review the simulation-based recommender system literature wherein the impact of recommendations on different kinds of diversity has been considered.

Agent-Based Modelling (ABM) has been applied in several works to understand the long-term impact of RS [2]. For instance, Zhang et al. [103] center their attention on simulating users’ consumption strategies, showing how, relying on recommendations, users may end up in the long-term contributing to the decrease of aggregate diversity. Zhou et al. [104] use ABM to study how preference bias—the distortion in users’ self-reported ratings caused by recommendations—influences the performance of recommender systems. In particular, they show how the bias introduced into the system through the users’ ratings may negatively influence the overall diversity of the recommended items.

Further examples of simulated environments can be found in the RS literature analyzing the impact of feedback loops. Mansoury et al. [59] design an iterative model to analyze the feedback loop, showing how it may cause a decline in aggregate diversity. Moreover, Jiang et al. [43] provide a theoretical analysis of the relationship between feedback loops, echo chambers, and filter bubbles. Instead, Chaney et al. [12] by simulating different models of users’ engagement with RS prove the impact of feedback loops on the homogenization of users’ behaviors. Similarly, Aridor et al. [6], by using numerical simulations to model user decision-making processes, provide an explanation of the findings of a previous study by Nguyen et al. [67] on the influence of recommender systems on content diversity. In the latter, the authors found that users’ interacting with the provided recommendations ended up consuming more diverse content in comparison to the users who did not. Aridor and colleagues confirm such results but also observe an increase in the homogeneity across users, i.e., decrease in aggregate diversity. Content diversity is also considered by Anderson et al. [5], observing a connection between recommendations and long-term reduction of diversity.

Under a different lens, sales diversity, defined as the concentration of consumer purchases, is at the center of attention in a series of studies by Fleder and Hosangar [24, 25] and Lee and Hosanagar [53, 54]. Fleder and Hosangar prove that collaborative filtering recommender systems exert a concentration bias, early delineating the idea that it may be possible that, at the individual level, diversity may increase, while at the aggregate level diversity may decrease. Lee and Hosanger, using field experiment data, confirm such results, highlighting how users do effectively explore novel items, thanks to the recommendations, but such explorations are highly correlated among users.

Another strain of simulation-based analysis focuses on comparing the performance of different typologies of recommender systems. Hazrati et al. [37] create a simulated environment where users are exposed to different kinds of recommender systems, proving that non-personalized methods produce the lowest diversity in terms of users’ choices. Jannach et al. [41] focus on the analysis of recommendation techniques using an iterative approach, wherein users are assumed to interact with a specific fraction of the recommended items. The authors show that, in terms of the spread and coverage of recommendations, several systems analyzed led to an increased concentration over time. Using the same methodology but specifically on session-based recommender systems, Ferraro et al. [23] find similar results in terms of spread and coverage.

A limitation of current impact-oriented RS research is the lack of an assemblage of a wide range of expertise. Indeed, being RS research mostly driven by computer science-inspired approaches, most of what until now has been measured as impact is the result of technical and engineering knowledge. Despite the several efforts to properly develop robust metrics and evaluation procedures, the narrowness of the considered approaches has limited the concept of impact itself to what is measurable according to such procedures, a limit that should be overcome to make algorithmic impact assessment practices shared and effective.

Several scholars in the music psychology and education field investigated the role of music exposure in influencing stereotypes and attitudes, starting from the idea that repeated exposure to a stimulus object may affect the preference towards such object, the so-called mere exposure effect theorized by Zajonc [102]. We highlight some examples hereafter.

Greitemeyer and colleagues [32] prove that exposure to music with pro-integration lyrics may reduce prejudice and discrimination towards immigrant groups, and later in Reference [31] they show that exposing listeners to music with pro-equality lyrics may enhance positive attitudes and behaviors toward women. In both cases, the authors found that the musical characteristics of the music exposed and the preference for it do not influence such impacts. Clarke et al. [14] investigate the relationship between music, empathy, and cultural understanding, showing that music exposure may indeed generate a sense of affiliation with unknown cultures. Their results are expanded in Reference [96], confirming that, especially in listeners with high trait empathy, music may increase positive implicit attitudes towards images representing members of foreign cultures. Tu [91] provides empirical evidence that exposing young students to 10 minutes of a Chinese music curriculum, when prolonged for 10 weeks, may impact the attitudes towards Chinese people. In a similar study, Sousa et al. [82] show that exposure to Cape Verdean songs together with Portuguese songs may reduce anti-dark-skinned stereotyping among light-skinned Portuguese children. Even if these examples consider different stimuli and different subjects, they all provide evidence that music listening may influence the idea that we have about other social groups, and more broadly about other cultures.

Another strain of research in the field of Music Psychology has been interested in understanding the impact of repeated exposure on music preference, based on Berlyne’s psychobiological theory (see Reference [13] for an overview). Among his several contributions, he theorized the existence of a relationship between aesthetic preferences and familiarity. Even if some studies support his claims, e.g., Reference [89], while others confute it, e.g., Reference [58], nowadays it is commonly accepted that familiarity with music has a prominent role in the development of preferences [44], a topic in which also neuroscience practitioners extensively debate [26]. In the Music RS literature, Sguerra et al. [80], performing large-scale analysis of users’ music consumption on Deezer, prove the validity of Berlyne’s theory by focusing on music discovery, showing how users’ interests may evolve in relation to the repeated exposure. These studies motivate our interest in exploring the impact that music recommendation diversity may have, connecting exposure, familiarity, and preferences. Indeed, while music recommender systems are known to be influential on exposure diversity [38, 73], little is known about how such exposure diversity may affect the users’ opinions, beliefs, and attitudes.

The Institutional Committee for Ethical Review of Projects (CIREP) at Universitat Pompeu Fabra approved the study design and confirmed the compliance of the research project with the data protection legal framework, namely, with the European General Data Protection Regulation (EU) 2016/679 (GDPR) and Spanish Organic Law 3/2018, of December 5th, on Protection of Personal Data and Guarantee of Digital Rights (LOPDGDD). A digital copy of the submitted documentation, the ethics certificate, and the data protection certificate are available upon request.

Before the main study, a smaller-scale pilot study was conducted to test the data collection process. The pilot study took place in February and April 2022, and the main study took place from May to July 2022. All participants were recruited using the online recruitment service Prolific,² and they were paid £6.00 per hour, the recommended minimum pay. They were informed about the voluntary nature of their participation, having the freedom to withdraw at any point, and of their rights including the right to access, rectify, and delete their information. They were also shown the information sheet describing the research objectives, methodology, risks, and benefits. Informed consent was obtained from all participants.

The prescreening of the participants was performed as a two-step process: first, using pre-determined criteria that are available in the recruiting platform (Prolific), and then, based on a questionnaire.

In the first step, we selected participants based on age (18–42), nationality and country of residence (Italy, Spain, and Portugal), fluent in English, who had participated in at least 20 surveys in Prolific, and with a task approval rate above 90%. In terms of gender, sex, and education level, no filter was applied. We chose to limit age including only Millennials and Generation Z subjects, i.e., those born between 1981 and 2012, known to have a predilection for EM [57, 68, 95], but also to narrow the generational differences among participants. The selection of only three countries in Southern Europe was motivated by the idea of having participants: (1) with a relatively similar cultural background; (2) living in the same time zone (GMT +1/+2). This last factor was fundamental to facilitating the daily interaction between participants and the researchers, the authors live in a country within the same time zone as the study participants.

Subjects matching the aforementioned criteria were redirected to the second part of the prescreening that consisted of a questionnaire in PsyToolkit [86, 87], a web-based framework to conduct psychological surveys and experiments. The questionnaire was composed of three main parts. In the first part, we asked participants to optionally confirm their demographic information. This step was included to double-check the reliability of the information provided by Prolific. In the second part, we asked for additional information about participants’ listening habits. In detail, they self-assessed with three 5-point Likert items: (1) taste variety, (2) EM listening frequency, and (3) EM taste variety. Additionally, we asked them to indicate the preferred music streaming platform and the average daily time spent listening to music. This information was used to filter out participants who (1) self-declared to listen to EM very often, (2) who indicated listening to music less than an hour a day, and (3) who did not select Spotify as the preferred streaming service. Even if including only Spotify’s users may be seen as a limitation in terms of generalizability of the results, this latter condition was necessary for collecting participants’ listening logs through ListenBrainz, as explained in the next section. The former two conditions were designed to create a group of subjects who listen to music more than occasionally, but who are not frequent EM listeners.

In the third part, we included a test to verify the participants’ familiarity with EM artists and genres. We replicated the test validated in a previous study [75], summarized hereafter. Participants had to specify whom from a list of mainstream EM artists was (i) known, (ii) possibly known, or (iii) unknown to them. The list was composed of 20 artists selected by AllMusic, an expert-curated online music database, as representatives of EM [3]. Afterwards, they had to do the same task for a list of EM genres, composed of 20 genres part of the Wikipedia page about EM [97]. The final score of each test, separately for artists and genres, was computed giving more points to participants who knew less popular artists (or genres). The rationale behind this is that knowing a popular EM artist or genre (such as Skrillex or dubstep) makes you less a connoisseur of EM at large, in comparison to knowing a less popular EM artist or genre (such as Autechre or IDM). The popularity of each artist and genre was computed using several signals from Spotify, Twitter, Facebook, Deezer, SoundCloud, and Last.fm, aggregated using the GAP0 metric proposed in Reference [50]. The list of artists and genres and the corresponding GAP0 score is presented in Appendix B (Figures 16 and 17). Afterward, we rank separately artists and genres by their GAP0 score, assigning then an alternative score inversely proportional to their popularity ranging from 1 for the less popular to \(1/20\) for the most popular. Participants’ familiarity score is computed as the weighted sum of the alternative scores, where participants got (i) the entire score if the artist/genre is known, (ii) half of the score if the artist/genre is maybe known, and (iii) zero if unknown. Finally, artist and genre familiarity scores are averaged, and participants who according to this test were too familiar with EM (i.e., average familiarity score \(\gt 5\)) were filtered out.

In summary, the following inclusion criteria were applied for the prescreening:

This prescreening allowed us to select a population of listeners quite homogenous in terms of demographics, listening habits, and familiarity with EM. Indeed, our main goal is to study the impact of EM recommendations on people who are not experts nor huge fans of this genre. We also aimed at reducing the response variability caused by different cultural backgrounds. These two aspects, familiarity with EM and cultural background, have been shown to be at the root of different perceptions of diversity in music lists [75], and by controlling for those while selecting the study participants, we aimed at minimizing the influence of such confounding factors in the analysis. The prescreening survey is accessible in the Supplementary Materials.

After being selected for participating in the study, participants were asked to create an account on ListenBrainz, allowing the collection of their listening logs for the entire duration of the study. ListenBrainz is a platform that keeps track of what music its users listen to and provides them with insights into their listening habits. It is operated by the MetaBrainz foundation,³ a non-profit organization that has been set up to build community-maintained databases and make them available in the public domain or under Creative Commons licences. Data is collected complying with the GDPR, and more information about MetaBrainz’s privacy policy can be found online.

Among the options for submitting the music listened to, it is possible to link ListenBrainz to the Spotify account. One of the advantages of this approach is the reliability of the metadata accessible for each log. Indeed, once a log is collected by ListenBrainz through its link with Spotify, the associated Spotify track ID and artist ID are available. With the retrieved IDs, by using the Spotify Web API,⁴ it is possible to obtain several types of data, from the acoustic properties of a track to the genres associated with the artists. However, this data collection method has some drawbacks.

First, creating a ListenBrainz account is a time-consuming task for which participants need to be paid, increasing the cost of the study. Moreover, people may be reluctant to link their ListenBrainz and Spotify accounts for privacy reasons. Last, while the use of the Spotify API is quite accepted in the Music IR and RS communities, the proprietary nature of the algorithms behind the API makes it difficult to know exactly how the data is generated. Nonetheless, by using ListenBrainz, we aimed to foster the reproducibility of our study, but also to ensure the availability of the collected data for future works making them publicly available through the ListenBrainz API.

The EMF questionnaire is designed to measure implicit and explicit attitudes towards EM. Participants completed it at the beginning, four times while being exposed to the music recommendations, and the last time at the end of the study, for a total of six times (see Figure 1). In particular, the EMF measures: (1) the participants’ openness in listening to EM; (2) the valence of their implicit association with EM; (3) the stereotypes they associated with EM. It is implemented in PsyToolkit, and the time needed to complete it is approximately 10 minutes. We now continue describing separately the three parts of the questionnaire.

Participants’ exposure to EM recommendations took place during the 20 daily listening sessions part of the COND stage. Being the sessions proposed on a daily basis, their design favored the easiness and rapidness of completing the proposed task, described as follows: As an initial step, a 30-second audio clip was presented to calibrate the audio volume to avoid exposing participants to extremely loud audio potentially damaging their hearing. Afterwards, we explicitly asked participants to entirely listen to a 3 minutes audio containing a mix of a few excerpts of EM tracks, asking them to be sure to be in a quiet environment and to allow themselves to be immersed in the music.

After each listening section, the participants provided their feedback on the music listened to. Specifically, they indicated with a 5-point Likert item if they liked or disliked the music, and they selected if the listened music was familiar or not. With that, the mandatory part of the listening session ended. Following, on a voluntary basis, they were asked if they wanted to explore the playlist with the full tracks part of the audio previously listened to. If selecting Yes, then they were redirected to a page with a link to a YouTube playlist. If they were not interested in discovering, then the listening session ended. The time needed to complete the mandatory part of each session was approximately 5 minutes. It is important to note that the interaction with the playlists was declared to be completely optional and did not affect the participants’ payment, i.e., they were not paid for the extra time spent interacting with the playlists. Figure 2 depicts the structure of a listening session.

This design allowed us to collect several types of data. First, the explicit liking and the familiarity ratings of the tracks listened to. Second, the willingness of discovering more about such tracks by choosing to explore the playlists. Third, by checking the YouTube playlist views, we also had a further metric of the actual interaction with the music proposed daily. While the definition of “legitimate view” in YouTube is not transparent [101], a common belief is that, to increase views, a user has to click the play button to begin the video, and the video has to be played for at least 30 seconds. Even if we cannot validate these hypotheses, we double-checked that simply accessing a YouTube playlist without listening to any tracks does not increase the view count of such playlists. We chose to avoid redirecting participants to Spotify playlists to not confound the listening log collections and the platform usage. Indeed, Spotify could have registered the signal of participants’ interaction with such playlists influencing eventually future recommendations by including EM related to the study. Last, YouTube playlists were not made public but accessible only to the study participants to avoid affecting the view count with interactions from external users.

Over the course of the study, we collected several types of feedback, implicit and explicit, quantitative and qualitative, that we used to understand the impact of music recommendation diversity. As the last step, we included a final survey to collect participants’ overall feedback on their participation.

The survey is composed of four sections. The first section (16 Likert items) includes questions about the participants’ relationship with EM before participating in the study. The second (16 items) collects feedback about the experience of participants during the study. In the third (21 items), we ask about participants’ feelings about EM after the study. Each of these sections is built to investigate the participants’ openness, appreciation, and willingness to discover EM. Besides, a few questions are related to stereotypes associated with this genre (e.g., EM has a fast tempo or it is mostly for partying), while others are about the perceived variety of the genre itself. Last, we insert a fourth section (8 items) to understand the overall impression of participating in the study. To avoid acquiescence bias, we balanced the number of positive and negative statements in each section, and items were presented in a randomized fashion to avoid order effect bias. The time needed to complete this survey is approximately 10 minutes. The complete list of items is presented in the Supplementary Materials.

We acknowledge that, as the questionnaire took place at the end of the study, participants may have not been able to exactly recall their relationship with EM before the study, and this could limit the reliability of quantitative measurements. Nevertheless, we include it to make participants reflect on their experience in participating in the study and on how such experience could have modified their vision of EM. Under this lens, we use this survey as a tool to gain a better understanding of the validity of the study and to improve future studies’ design.

The first step was to create a dataset of candidate EM tracks to be included in the listening sessions. The goal was to select a set of tracks covering as many EM styles as possible to create a varied representation of the EM culture, however, without the presumption of including every existing nuance of it. We consulted Wikipedia [97] and Every Noise at Once⁶ (ENaO) to select 20 EM genres with associated 165 subgenres, listed in Table 7 (Appendix B). For each of the subgenres, we retrieved from ENaO a playlist of around 100 representative tracks for a total of around 16k tracks. Then, we filtered out tracks too popular by using the Spotify popularity indicator and the YouTube view count to avoid that familiarity with the tracks could have an effect on participants’ ratings. Last, we randomly select 10 tracks (when available, if not less) for each subgenre, remaining with a final set of 1444 candidate tracks.

As a second step, we made use of Music Information Retrieval (MIR) techniques to (1) extract audio embedding from the candidate tracks using state-of-the-art Deep Learning models; (2) validate these models by using standard MIR hand-crafted features. Tracks’ audio embeddings were extracted using EfficientNet [90] trained with a dataset of tracks annotated with Discogs⁷ metadata. Among the models tested, this showed the best performance in terms of clustering the candidate tracks coherently with respect to the considered taxonomy of EM genres. Figure 13 (Appendix B) displays a 2-dimensional representation of the embedded space obtained with this model. Then, focusing on four features (tempo, danceability, acousticness, and instrumentalness), we investigated the consistency of the tracks’ embedded space. In particular, we centered our attention on if track embeddings clustered together displayed similarity also in terms of the aforementioned features.

The diversification process for creating the recommendation lists to which participants were exposed during the study was based on two main criteria. First, we aimed at controlling the inter-list diversity to present to one group of participants a varied selection of EM throughout the 20 listening sessions, while to the other group only a tiny fraction of EM. Second, we limit the intra-list diversity for both groups to avoid creating listening sessions too varied and fragmented to become potentially annoying. Having study participants not familiar with EM, to ask them to listen in sequence, e.g., to a glitchy IDM track and then a soft Minimal Techno track, could have negatively impacted their listening experience, consequently affecting the drop-out rate.

Other design choices of the recommendations were the following: First, each list had to be formed by four tracks to limit the length of the listening session. Second, each list had to contain tracks with no more than three different genres. Indeed, even if the tracks’ embedding preserves some musical characteristics of EM genres (see Appendix A), it is also true that two tracks could be very near in the embedded space even if labelled differently.

Having in mind these aspects, we implemented the following strategies: To minimize the intra-list diversity, we found for every track in the dataset the three nearest tracks, according to the pairwise cosine distance between embeddings. These quadruples of tracks were the candidate recommendation lists. Afterwards, to minimize the inter-list diversity, we selected 20 quadruples from a single genre (Figure 3(A)), while, to maximize it, we selected one quadruple for each of the 20 genres in the dataset (Figure 3(B)).

We selected Trance as a seed genre of recommendation lists with low inter-list diversity (from now on, simply low diversity or LD) for the following reasons: First, it has a number of subgenres that ensure some variability between lists, without exceeding like in the case of House (see Table 7). With regard to this aspect, further valid choices could have been Techno, Ambient, or Hardcore. However, the Trance tracks’ cluster has, on average, a higher silhouette score (Figure 12), meaning that in the embedded space, tracks were better clustered than the other genres. This aspect was important to ensure to have enough candidates for creating coherent listening sessions with low intra-list diversity. Second, Trance music reflects some stereotypes of electronic music: quite danceable, with almost no use of acoustic instruments and a small presence of vocal parts. Hence, participants exposed to LD recommendations interacted with a stereotypical idea of the EM culture, with few variations in terms of musical properties. Even if differences between sessions existed because of different characteristics of the Trance subgenres, a certain level of homogeneity was ensured by having selected a single genre to create the recommendations. It is important to note that the use of a genre different from Trance could have affected the results, especially in terms of participation and engagement in the study. Indeed, Trance music reflects few stereotypes associated with EM, meaning that its sounds are somehow expected also for people not familiar with this genre. We believe that using a more sophisticated and less popular genre, e.g., Intelligent Dance Music (IDM), could have made it more difficult for the participants to go through 20 listening sessions, consequently increasing the drop-out rate.

On the contrary, to obtain high inter-list diversity (from now on, simply high diversity or HD), we picked a quadruple of tracks for each different genre, which was enough to ensure that participants exposed to such recommendations would explore different facets of EM. These two diversification strategies effectively led to statistically significant differences between recommendation lists. In detail, we tested these design objectives:

According to the statistics presented in Table 2, the aforementioned design objectives were satisfied. The objectives are further validated by the boxplot presented in Figure 18 (Appendix B), which shows the features’ distribution.

Thanks to the described procedure, we created recommendation lists formed by 4 tracks, 20 with high and 20 with low diversity. In every list, tracks were selected to minimize their differences according to the distance between their embeddings. The final step was to mix the four tracks in each recommendation list to create a single audio file to be included in a listening session. To accomplish that, we implemented the following procedure using Pysox, a Python wrapper around Sox [8]: First, we randomly selected a 45-second excerpt for each track in the list. The sample rate of each excerpt was converted to 48,000 Hz and the volume normalized to \(-\)3db. Then, we joined the excerpts together, including a 1-second fade-in and fade-out, to help the listeners recognize the start and the end of each track in the mix. At the end of this process, we obtained a 3-minute mp3 audio for every recommendation list.

A final manual check of each audio was done to ensure that each listening session was properly mixed. We additionally verified that the content in each audio was appropriate for the purpose of the study. Indeed, it is not uncommon that lyrics in EM could contain references to sex, use of drugs, or blasphemy. While we do not want to advocate for a moral judgment of the artists’ forms of expression, still we agreed that it was necessary to remove some tracks to respect every subjectivity that might eventually participate in the study.

The exploratory nature of the study, and the lack of any meta-analysis that defines the ground truths of our variables, made it difficult to estimate with a power analysis the exact number of participants needed to observe potentially existing statistical differences. Nonetheless, guidelines from Human-Computer Interaction research helped us in estimating a valid number of participants [11, 15, 45]. Indeed, when performing a t-test (or an equivalent non-parametric statistical hypothesis test) for the difference between two independent means, to observe a medium effect size (an effect likely to be visible to the naked eye of a careful observer), at a significance level of \(\alpha =.10\) (acceptable for an exploratory study), with a statistical power of .80 (to avoid incurring a too-great risk of a Type II error), according to Cohen [15], a sample of size 100 participants is required, i.e., 50 participants for each group. This is the reason why, following the prescreening, we recruited 110 participants for this study, also foreseeing the eventuality of some of them dropping out.

Most of the selected participants are aged between 18 and 32 years (91%), come from Portugal (61%), Italy (32%), or Spain (7%), almost equally divided into binary genders (53% women and 44% men)⁸ with a small fraction of non-binary participants (3%). In terms of education, 69% have a bachelor’s degree or lower, and 63% of the participants are still enrolled on a study course (bachelor, master, etc.). According to their self-declared answers, 88% affirm having varied listening habits, only a quarter affirms listening often to EM, and a third affirms listening to a varied selection of EM. In terms of listening time, 78% declare to daily listening, on average, between 1 and 3 hours of music.

The two familiarity tests (artist and genre; see Section 3) help us in estimating the participants’ knowledge of the EM scene. The test score ranges from 0, if no item in the lists is known, to 10.5 if all items are known. In the artist test, the average score obtained by the selected participants (\(N=110\)) is 1.7 \(\pm\) 1.2. Instead, the average score of all the participants in the prescreening (\(N=437\)) is 2.6 \(\pm\) 2.1. In terms of genre familiarity, the average score is 5.1 \(\pm\) 1.6 against 5.7 \(\pm\) 1.9. Averaging over the two tests, the recruited participants got 3.4 \(\pm\) 1.1 against 4.2 \(\pm\) 1.8. As these numbers evidence, the filter applied during the prescreening made it possible to select a group of participants, on average, less familiar with mainstream EM artists and genres in comparison to a wider group of listeners with shared demographics. In the next section, we further characterize the study participants by analyzing their listening logs.

After selecting the participants matching our prescreening criteria, we needed to split them into two groups, one to be exposed to recommendations with high diversity and one with low diversity. During the pilot study, we realized that, given the size of the sample, randomly assigning participants to groups could result in an unbalanced baseline for the variables we were interested in studying. For instance, one group could have been formed by participants much more open to listening to EM than the other group. To avoid imbalance between characteristics, which may have affected the impact of the recommendations, we assigned participants using covariate adaptive randomization [88], creating two groups balanced in terms of familiarity with EM (familiarity test score), openness in listening to EM (o-score), implicit association with EM (d-score), and the number of EM tracks listened to during the PRE stage. At the end of this process, we got a group of 55 participants assigned to the HD group and a group of 55 participants assigned to the LD group.

Table 3 summarizes the participation of the two groups in the EMF questionnaire and in the listening sessions. Not surprisingly, participation decreased over the course of the study, notably between the PRE and POST phase in the EMF questionnaire (\(-\)19%) and less between the first and fourth week of the listening sessions (\(-\)4%). Overall, we may notice a small difference between the two groups, with the LD participants being more active during the study than their HD counterparts. Some participants have been excluded during the course of the study if (a) they never showed up after being selected (initial nonresponse) or (b) after a certain point they stop participating (attrition). The initial nonresponse was quite high for the EMF questionnaire (HD: 7%, LD: 5%), but relatively small for the listening sessions (HD: 5%, LD: 0%). Instead, attrition was equal for both tasks (HD: 7%, LD: 2%). These numbers are in line with retention rates observed in Prolific and generally in longitudinal studies [17, 49].

In conclusion, we excluded from the analysis the responses of the participants who did not participate in: (a) more than 4 listening sessions and (b) more than 3 EMF questionnaires. With this choice, we ended up analyzing the responses of 94 over 110 participants (85%), 45 over 55 in the HD group (82%), and 49 over 55 in the LD group (89%).

During the 20 listening sessions in the COND stage, we collected four types of data to measure the impact that recommendations had on the participants of the HD and LD groups:

Figure 5 displays the distribution of playlists’ accesses and interactions, where trend lines are computed with the linear least squares method. Similarly to the overall decrease in study participation, we may note that over the course of the sessions the overall engagement with the playlists decreased. Looking at the number of accesses and interactions, the HD participants seem to be more interested in discovering the music listened to than the LD ones. Moreover, we notice that on several sessions LD participants accessed a playlist but had zero interactions with it (e.g., sessions 10 and 11). This phenomenon never occurred to HD participants, who, on the contrary, in some sessions had much more interactions than accesses (e.g., sessions 5 and 17), meaning that some participants interacted several times with the same playlist. Naturally, in both groups, accesses and interactions are positively correlated (HD: \(\rho =.75\), LD: \(\rho =.51\)).

Figures 6 and 7 merge the playlists’ data and the like ratings, giving us an alternative view for understanding the participants’ reception of the recommendations. In Figure 6, it emerges that the HD group disliked more sessions and with more extreme ratings compared to the LD group, which, on average, seem to have appreciated most of the music they have been exposed to. Nevertheless, HD participants, even when they mostly disliked a session, interact with the playlists on YouTube (e.g., sessions 4 and 17).

Such behavior is confirmed in Figure 7, where like ratings are split between participants who accessed the playlists (light bars) and those who did not (dark bars). We see that some of the HD participants choose to access the playlist and discover more about the tracks listened to even in the most disliked sessions (negative light bars), a behavior not present for the LD group. On the contrary, some of the LD participants, even if liking the tracks listened to, choose to not interact with the playlists (positive bars with zero interactions, e.g., Figure 6, sessions 10 and 11) or not access the playlists (positive dark bars, e.g., Figure 7, sessions 4 and 10).

The familiarity ratings’ results are shown in Appendix B (Figures 21 and 22) and summarized as follows: The sum of ratings is negative for almost every session, indicating that the tracks were mostly unfamiliar to the participants.¹⁰ This trend was expected because of participants’ non-familiarity with EM, but also because of the popularity filter applied before creating the recommendations. Besides, we observe a positive correlation between like and familiarity ratings (HD: \(\rho =.55\), LD: \(\rho =.72\)), confirming what previous music psychology scholars have extensively proven: The more familiar a track sounds, the more is likely to be appreciated.

We further confirm the previous findings by performing Mann-Whitney U tests comparing the playlist accesses and interactions, and the like and familiarity ratings of the HD and LD groups’ participants, aggregated by listening session. This test is commonly used to compare the differences between two independent samples when the sample distributions are not normally distributed and the sample sizes are small (in this case, the sample size is the number of listening sessions \(n=20\)). In particular, we verify that:

Table 4 reports the outcomes of the tests. (1) and (2) are confirmed by a significant difference between HD and LD groups in terms of playlist accesses and interactions. In terms of like ratings (3), we see a smaller effect size with only 62% of sessions having HD group ratings lower than the LD ones. This proportion increases to 70% if looking only at the ratings of participants who did not access the playlists (4). This confirms that even when LD participants liked the tracks in the listening sessions, they interacted with the playlists less than the HD group. Last, looking at the familiarity ratings (5) no significant difference is found, confirming that the design of the listening session was effective, exposing subjects to music they were mostly unfamiliar with.

Through the analysis of the listening sessions, we have shown the distinct reactions of the two groups of participants when receiving the music recommendations. Hereinafter, we continue by focusing on the impact of the exposure to EM recommendations, first, in terms of openness in listening and implicit association, and second, in terms of stereotypes that participants associated with this music genre.

The analysis of the EoS survey reveals a few more qualitative insights that complement what is presented in the previous sections. First, we analyze participants’ answers concerning the openness and appreciation of EM before, during, and after participating in the study. Then, we focus on a set of stereotypes to see how exposure to EM recommendations has affected the participants’ opinions.

We recall that the survey is formed by three main groups of items, the first asking about the experience of participants before the study, the second during, and the third after the study. The survey contains 16 Likert items to measure the participants’ openness in listening to EM and 16 items for measuring appreciation of EM, divided into 6 items asking for participants’ beliefs before, 6 during, and 4 after participating in the study. By analyzing these three sets separately, we may get an idea about how participants perceived a change in their openness and appreciation of EM due to their participation in the study.

Figure 10 presents the distribution of the responses for the whole group of participants. We avoid reporting results separately for the HD and LD groups, because no significant difference has been found between their responses. Analyzing the internal consistency of the sets of items computing Cronbach’s alpha (\(\alpha\)), we observe an acceptable consistency in the three sets. Indeed, for the openness items, we have: before \(\alpha =.88\), during \(\alpha =.82\), and after \(\alpha =.72\). For the appreciation items: before \(\alpha =.90\), during \(\alpha =.86\), and after \(\alpha =.77\). Therefore, we may assume that the items properly reflect the two concepts that we aimed at measuring.

Focusing on the openness in listening to EM, we notice that participants’ responses before the study were quite balanced around the neutral response, even if with high variance (\(M=2.94\) \(\pm\) 1.31). When asked about their openness during and after the study, they averagely declared to be more open in comparison with the beginning (respectively, with \(M=3.43\pm 0.95\), and \(M=3.48\pm 1.08\)). Therefore, their perceptions are in line with the results of Section 5.4, wherein an overall increase in openness was found by looking at the results obtained from the Guttman scale. In terms of appreciation, we notice a similar trend with an initial balanced situation at the beginning of the study (\(M=3.07\pm 1.27\)) that increases towards more positive responses over the 12 weeks (\(M=3.36\pm 0.99\) and \(M=3.67\pm 1.01\), respectively, during and after the study).

The second focus of our analysis is on comparing four items describing four stereotypes of EM, reported in Figure 11 together with the distribution of the responses. In this case, it displays the distribution for the HD and LD groups separately, because the impact of the recommendations on these items has indeed been mediated by their diversity.

For the first item analyzed (“I listen to EM only for partying”), we observe that, prior to the study, participants had, on average, a balanced response towards this stereotype (HD: \(2.98\pm 1.06\), LD: \(3.17\pm 0.97\)). However, during the study, the two groups realize that may be restrictive to consider EM only for partying, with an overall decrease in their average response (HD: 2.47 \(\pm\) 1.03, LD: 2.56 \(\pm\) 1.07). In the case of the second item (“I believe Electronic Music has mostly fast tempo and high energy”), we observe a similar behavior. In the beginning, participants strongly agree with this statement (HD: 4.26 \(\pm\) 0.66, LD: 4.15 \(\pm\) 0.68), but then after the study they changed their opinion (HD: 3.51 \(\pm\) 1.1, LD: 3.69 \(\pm\) 0.83), agreeing less strongly to the fact that EM has a mostly fast tempo and high energy.

The next considered item (“I think of Electronic Music as a varied genre with several sub-styles”) presents an opposite situation in comparison to the former two. Indeed, at the beginning, on average, participants neither agree nor disagree significantly with the statement (HD: 3.02 \(\pm\) 1.12, LD: 3.19 \(\pm\) 1.0). However, participating in the study made them change their opinion about it, agreeing much more in percentage than before (HD: 4.21 \(\pm\) 0.77, LD: 3.85 \(\pm\) 0.77). Similarly, also in the case of the fourth item considered (“I believe that Electronic Music could fit in different contexts”), we observe a neutral position of participants at the beginning (HD: 3.37 \(\pm\) 1.05, LD: 3.35 \(\pm\) 0.9), which later agree more with the fact that EM may fit in different contexts (HD: 4.12 \(\pm\) 0.73, LD: 3.83 \(\pm\) 0.72). It is interesting to note that the responses in these two latter items are the only ones for which we found a significant difference between HD and LD subjects. Indeed, by performing a Mann-Whitney U test, we obtain a p-value of .01 and .02 and a CLES of .65 and .62. This indicates that, even if for both groups the exposure to recommendation affected the beliefs about how varied EM may be and how it may fit in different contexts, in the case of the HD group, this shift has been more pronounced.

The most pronounced role that recommendation diversity plays in our study is with respect to the curiosity generated in the participants when exposed to music during the listening sessions. Indeed, the listeners’ willingness to explore EM playlists is significantly higher if exposed to highly diverse recommendations. Having in mind that EM was mostly unfamiliar to the subjects in our study, we believe that when listeners are not familiar with a music genre, a diversified set of recommendations could increase their willingness in exploring such music. A message sent to us by a study participant supports this hypothesis: “Hello! It was a pleasure participating, I discovered new artists that I really liked! Happy to have contributed to your study :) Thank you!”

Limitations. Nevertheless, the first limitation of this work rises from the definition of discovery itself. Indeed, if, as Nowak suggests, discoveries are “affective responses to music content that occur within individuals’ life narratives and mediate their interpretation and definition of music” [69], then we argue that the responses resulting from the participation in our study cannot be compared to what listeners usually experience in less artificial situations, wherein they are exposed to music. Moreover, focusing on discovery strategies and behavioral attitudes, Garcia-Gathright and colleagues [27] show how explorative goals may vary according to the listeners’ needs. Under this lens, a further step could be to link the various overt behaviors to people’s affective responses to evaluate if the discovery mediated by algorithmic recommendations has some shared values with other non-digital forms of curation (e.g., a DJ tracklist created for a live event).

Throughout the study, participants’ openness to listening to EM increased, indicating that, in the long term, the exposure to unfamiliar music could favor listeners in being less reluctant in approaching a genre previously unknown. This finding does not come totally anew, partly confirming results from the literature on the impact of repeated exposure to music.

Diversity here seems to motivate particularly participants who started the experiment affirming to be not open to listening to EM. Indeed, subjects who passed from not being open to being open to listening to EM for one hour or more a week are participants in the HD group who passed from not being open to being open to listening to EM for one hour or more a week are two times the ones of the LD group. Still, the impact of recommendation diversity is apparently not significant. Moreover, contrary to the implicit association, participants’ openness was more consistently affected right after the conditioning stage than at the end of the study. Connecting this with the impact on discovery, we deduce that openness is highly influenced by exposure, but such influence decays rapidly when listeners stop to be exposed to recommendations. In light of this, we support the idea that when listeners are not familiar with a music genre, repeated exposure to recommendations could increase their openness in listening to such music. A message sent by one of the participants of the HD group right after the end of the COND stage supports this intuition: “Thank you so much [...] It was very interesting and a good opportunity to learn about this musical genre.”

Limitations. Again, the settings of the experiment may have influenced the outcome. In fact, participants were requested to be highly focused while listening to the music during the study, a condition that is not quite frequent in today’s listening practices, where music is often confined to the background while performing other activities [64]. This could have affected the openness more consistently in comparison if tracks would have been passively listened to, for instance, while walking in a mall or while working. Repeating the experiment in a non-online environment may lead to more precise results in this regard.

Furthermore, the use of the Guttman scale has its own limitations. Indeed, listeners’ openness in listening to EM could be determined by an overall curiosity in discovering music. For instance, a listener with very heterogeneous musical tastes could be open to listening every day to one hour of electronic music, one hour of classical music, one hour of rock music, and more. On the contrary, a very homogeneous listener would avoid listening to electronic, classical, and rock music at the same time if disliked. We foresee the analysis of participants’ openness in listening to EM with regard to their overall tendency of listening to varied music.

Nevertheless, even if the concepts of openness in listening and discovery may overlap, we believe they do represent two different attitudes. The willingness to discover unfamiliar music for the sake of curiosity does not translate automatically into being open to listening to it. In fact, discovery might eventually lead to being less open. On the contrary, someone could be open to listening to some kind of music without being willing to discover something new. In conclusion, we believe that the willingness to explore could be related to people’s openness in listening, and future work should explore the relationships between these two attitudes while listening to music.

The exposure to music recommendations helped participants in deconstructing part of the preexisting positive or negative association to EM, developing a more neutral attitude during the 12 weeks. The impact of diversity in this case is not significant, considering that no major differences have been found between the two groups. In fact, we argue that the tendency of the HD group to decrease more significantly in comparison to the LD group is because of the unbalance created by the different dropout rates.

Based on that, we hypothesize that when listeners are not familiar with a music genre, to receive repeated recommendations could mitigate the valence of the implicit association with such music. The results obtained deviate from the findings in Reference [96], where positive implicit attitudes towards facial images of people from two cultures, namely, Indian and West African, are developed by mere exposure to music from such cultures. However, the types of association and stimuli that listeners experienced in our study are undoubtedly different and presented in a different scenario.

Limitations. The following reasons may be at the root of the development of neutral attitudes towards EM in the presented study: First, and in line with the previous point on the discovery, the experimental setting mediated the affective response, also influencing the implicit association. The tendency of the participants to associate neutral valence with EM genres could be motivated by the artificiality of the events wherein participants listened to the music. Second, given the participants’ unfamiliarity with the genre, it is understandable that they did not develop any positive or negative attachment to genre labels that they might have never heard of before the study.

As elegantly discussed by McLeod [61], genre namings are strictly tied to the EM communities behind which people identify, also influencing the dynamics of group formation. Therefore, we hypothesize that most of the genres shown in the SC-IAT test were only labels to the study participants and remained as such, given that no mechanism of bounding was incentivized. Indeed, no information about the genre of the tracks listened to was provided during or after the listening sessions. Under this lens, receiving neither positive nor negative responses from the participants may be seen as the desired result in the long term, which, however, needs further analysis to be understood in depth.

What emerges from the analysis of the results is that the idea that participants have of EM reflects stereotypes usually associated with this music genre. Indeed, listeners not familiar with EM may fall into the trap of misinterpreting it as music composed only with electronic instruments (e.g., drum machine, sampler, synthesizer). Instead, acoustic instruments, even if sampled, filtered, or generally modified, have always been used by EM artists. Under this lens, the fact that acousticness is the only feature on which participants’ changed their idea after the exposure to recommendations does not come as a surprise.

Moreover, the stereotype that EM is only for parties and the common belief that it has mostly a fast tempo and high energy have been partly deconstructed by the exposure to music such as Ambient, Electroacoustic, or Chill-out. Nevertheless, the characterization of EM as Energetic and Rhythmic is quite common also in scientific literature, e.g., Reference [77], and participants of the LD group at some level experienced this aspect of EM. Besides, another accomplishment of this study is to show that listeners exposed to diversified recommendations may have a better understanding of the variety of EM culture and realize that this music may fit in different contexts, not only for partying. Therefore, we deduce that exposing listeners not familiar with a music genre to a set of diversified recommendations, showing the different facets of such genre, could deconstruct pre-existing stereotypes. This finding is in line with the music psychology literature on the influence of music exposure.

Limitations. Likewise, the representation that participants have of EM artists is quite stereotypical, and also in this case it was something expected. EM artists are, according to them, mostly men, white, under 40, and coming from developed and high-income countries. In this case, we did not expect that recommendations would affect participants’ opinions, mostly for two reasons, which however, are also two limitations of this work. First, participants’ origin has been purposely restricted to a small part of the world, normally labelled as Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. Similarly, also the music material part of the study is somehow biased towards Western artists. In fact, the semi-automatic method for creating the dataset has produced recommendation lists that reflect the variety of EM as a music genre, but not its variety as music played in different regions of the world.

One of the goals of the study was to expose people not familiar with the EM culture to different genres that could fit under this label. Therefore, our aim was to select as many as possible varied tracks to represent the richness of this culture, even if we did not aim to create a dataset representing its entirety. To do that, we designed a semi-automatic method based on two sources: Wikipedia and Every Noise at Once.¹⁴

The initial step consisted in retrieving the list of EM genres from the Wikipedia page “List of electronic music genres” [97]. While several taxonomies and hierarchies of EM genres and subgenres have been published, we believe that the information on Wikipedia properly represents the variety of EM, enough precisely for our purposes. From that source, we obtained a set of 20 genres and 320 subgenres.¹⁵ The second step was to map these subgenres to the ones part of the Every Noise at Once (ENaO) website. ENaO is described by its creator Glenn Mcdonald as “an ongoing attempt at an algorithmically-generated, readability-adjusted scatter-plot of the musical genre-space, based on data tracked and analysed for several genre-shaped distinctions by Spotify” [83]. Born as a debugging tool, in its current form, the website presents for each genre label a playlist formed by approximately 100 tracks, considered representative of the genre by Spotify algorithms. Mapping Wikipedia to ENaO, we linked to the 20 genres only 181 subgenres, for which we retrieved the corresponding playlists.

A few aspects of the aforementioned approach should be noted: First, even if we are aware of the dynamical, intrinsic, ambiguous, and context-dependent nature of concepts such as genre and style [73], approaching music cultures as broad as Electronic Music is quite natural to identify different aesthetic and social characteristics in its several subgenres [61]. For instance, even without knowing anything about EM genres, it is possible to recognize the differences between the fast breakbeats of the Drum and Bass and the slow-tempo beats of Downtempo. As a matter of fact, in a previous study, we showed how familiarity with styles and subgenres highly influence the perceived diversity of people exposed to EM tracks [74]. Even if it could have left out some genres, pursuing a bottom-down approach to find representative Electronic Music, starting from genre labels arriving at tracks, seems to us the most obvious approach to creating a varied EM dataset.

Second, the choice of using ENaO playlists to select candidate tracks could be criticized because of the opaqueness of the Spotify algorithms. Indeed, it is not possible to find the exact description of the procedure that assigns a genre label to artists and albums. Even having in mind this limitation, after exploring in depth the ENaO website, we believe that it provides a good representation of the genres on its map, sharing the idea of ENaO creator that “the point of the map, as with the genres, is not to resolve disputes but to invite you to explore music” [60]. Under this lens, we do not argue that the classification in this study is the ultimate one, but one of the possible classifications that may help listeners navigate the EM culture.

After a preliminary exploration of the dataset, we performed a manual cleaning of some of the music selected to be sure that crossover genres would not enter into the final listening sessions. We do not argue if Electronic Rock may be considered Electronic music, Rock music, or both, but we believe that crossover music did not fit the purpose of our study. In the end, we restricted the dataset to 20 genres (ambient, bass music, breakbeat, chill out, disco, drum and bass, electroacoustic, electronica, garage, hardcore, hardstyle, hauntology, house, IDM, jungle, noise, plunderphonics, techno, trance, videogame) and 165 subgenres listed in Appedix B (Table 7). At that point, we had a pool of around 16k tracks (\(\sim\)100 tracks for subgenre) listenable on Spotify.

Further filtering was applied by looking at the popularity of the tracks. Indeed, several studies have proven the influence of familiarity on music preferences [13]. To avoid creating a popularity effect in our study, i.e., participants’ ratings influenced by the popularity of a track, we filtered the dataset by using two indicators: (1) Spotify track’ popularity; (2) YouTube view count. In detail, we implement the following procedure: From the 16k tracks for which we already had the Spotify ID, we filtered out the ones with popularity less than the first quartile Q1 and major than the third quartile Q3 computed on the Spotify popularity indicator. For the remaining ones, we made use of a Python wrapper to search with the YouTube API for the corresponding video by using the track names. As a result, we obtain a total of approximately 8k tracks with Spotify ID and YouTube ID, already filtered by Spotify popularity. The last step was to further filter out tracks according to the YouTube view count applying the same logic of Spotify popularity, including only tracks with views between Q1 and Q3. After the second filtering, we randomly chose 10 tracks for each subgenre obtaining 1,444 candidate tracks, for which, finally, we extracted the audio embedding, as reported in the next section.

The use of deep representation models in MIR research is widespread and applied in several retrieval and classification tasks, e.g., auto-tagging, instrument recognition, genre classification, and ultimately music recommendations [100]. Nevertheless, the trustworthiness of such representations is still under the scrutiny of the research community, partly because of the still low interpretability in comparison to traditional hand-crafted music representations, usually informed by human music domain knowledge [48]. Even if the goal of this study was not to perform a rigorous comparative analysis of different music representation models, we were nevertheless interested in creating diversity-aware content-based music recommendations, on the one hand, based on state-of-the-art deep learning models, and on the other hand, interpretable enough to make us aware of the characteristics underlying the diversification outcomes. Consequently, we first started exploring several deep representation models, and then we attempted to interpret those by using a set of hand-crafted features.