ARAUS: A Large-Scale Dataset and
Baseline Models of Affective Responses to
Augmented Urban Soundscapes

The largest curated audio dataset is arguably AudioSet, which at the time of its initial release contained 1.7M 10-second segments of audio from YouTube videos organized in a systematic ontology for audio tagging [25]. Strong labels have subsequently been provided for a subset of audio [26], and at the time of writing, AudioSet has grown to about 2.1M segments, with 120K being strongly labeled [27]. However, the publicly available version of the data constituting AudioSet is in the form of $128$-dimensional VGGish features [28] due to potential copyright issues related to the use of raw audio from YouTube videos, which limits its use as a public dataset.

Alternatives to AudioSet include UrbanSound8K [29], ESC-50 [30], and FSD50K [31], which contain labelled Creative Commons-licensed tracks from Freesound [32] and serve as publicly-available benchmark datasets for weak audio classification. However, the nature of the stimuli in these datasets tends to be monophonic or same-class polyphonic. While this is useful in reducing the complexity and noise in inputs to train robust sound event classification models, the full complexities of real-life acoustic environments necessary for soundscape research are rarely represented in these datasets. Individual monophonic stimuli may be used to compose synthetic soundscapes like in the URBAN-SED dataset [33], which used the tracks in UrbanSound8K, but the focus of UrbanSound8K on just 10 possible event classes may be insufficient to emulate the variety of sound sources possible in real-life urban environments.

Therefore, multiple efforts have been made to record real-life acoustic environments, which are generally polyphonic in nature, and provide them as publicly available datasets for use in sound and soundscape research. These include Urban Soundscapes of the World (USotW) [34]; EigenScape [35]; SONYC Urban Sound Tagging (SONYC-UST) [36] and SONYC-UST-V2 [37]; TUT Acoustic Scenes [38, 39] and TAU Urban Acoustic Scenes (TAU-UAS) [40, 41, 42]; Singapore Polyphonic Urban Audio (SINGA:PURA) [43]; Ambisonics Recordings of Typical Environments (ARTE) [44]; and Sony-TAu Realistic Spatial Soundscapes 2022 (STARSS22) [45].

However, the aforementioned datasets contain no corresponding “subjective” labels concerning the affective perception of the recorded environments. This limits their use as datasets for soundscape augmentation, because knowing how individual soundscapes are perceived by humans is crucial in analyzing and modeling their perception. In addition, with the exception of USotW, the recordings were not compliant with ISO/TS 12913-2:2018 [20] due to them preceding the publication of the standard, or being made for a different purpose. Hence, they may not be immediately suitable for use under the ISO 12913 paradigm, which the ARAUS dataset was designed under.

Nonetheless, audio datasets with labels specific to perceptual indicators of the stimuli also exist. For example, the International Affective Digitized Sounds (IADS) dataset [46] has had labels for discrete emotional categories elicited by the $167$ individual stimuli provided by [47], with an expanded version (IADS-E) provided by [48]. The largely monophonic stimuli in IADS, however, suffer from the same drawbacks as datasets based on Freesound. As a workaround, the Emo-Soundscapes dataset [49] used individual clips from Freesound to synthetically generate $1213$ soundscapes, each $6\text{\,}\mathrm{s}$ long, and obtained corresponding valence-arousal labels based on the Self-Assessment Manikin (SAM) [50] from participants on the CrowdFlower platform.

In contrast, datasets with perceptual labels for real-life audio recordings include the Athens Urban Soundscape (ATHUS) dataset [51], which contains $978$ crowd-sourced recordings with corresponding labels for subjectively-rated soundscape quality on a five-point Likert scale, as well as the International Soundscape Database (ISD) [52], which as of v0.2.1, consists of $1258$ $30$-second long recordings in $13$ European cities and corresponding perceptual responses collected using the Soundscape Indices (SSID) Protocol [53]. The perceptual responses collected using the SSID Protocol were largely inspired by the Method A questionnaire in ISO/TS 12913-2:2018 [20].

However, the relatively smaller sample sizes of these datasets, as compared to large-scale datasets like AudioSet, may preclude their use in developing high-parameter models, such as deep neural networks, which have shown state-of-the-art performance in various “objective” acoustic tasks. Hence, the ARAUS dataset was designed at a relatively large scale to be amenable to high-parameter models.

For the five-fold cross-validation set, all base urban soundscapes were taken from the Urban Soundscapes of the World (USotW) database [34]. The USotW database contains $127$ publicly available recordings of urban soundscapes covering an extensive variety of urban environments from various cities around the world. The urban environments range from parks to busy streets, and the cities include those in Asia, Europe, and North America, which allows the ARAUS dataset to be broad-ranging in its coverage of real-life urban soundscapes. The urban soundscapes in the USotW dataset were also chosen by the USotW team via clustering of locations reported by local experts to be “full of life and exciting”, “chaotic and restless”, “calm and tranquil”, and “lifeless and boring”, which are adjectives spanning the perceptual space generated by the “Pleasantness” and “Eventfulness” axes of the ISO/TS 12913-3:2019 circumplex model [11]. This heightens the suitability of the USotW database for use in the investigation of the affective qualities of soundscapes that the ARAUS dataset aims to enable.

Each $60$-second binaural recording in the USotW database was split into two halves of $30\text{\,}\mathrm{s}$ for the creation of the audio-visual stimuli in the ARAUS dataset. Consequently, the audio-visual stimuli in the ARAUS dataset are all $30\text{\,}\mathrm{s}$ in length, which is in line with the stimulus length used in several soundscape studies, such as those in [54, 55, 56]. Upon splitting the recordings into two halves, we discarded any half with (1) audible electrical noise (such as those caused by a faulty microphone or loose connection), in order to reflect only accurately-captured real-life soundscapes; (2) measured in-situ $L_{\text{A,eq}}$ values below $52\text{\,}\mathrm{dB}$, in order to ensure that reproduction levels were significantly above the noise floor of the laboratory location with the highest noise floor (about $37\text{\,}\mathrm{dB}$; see Fig. 3) where the subjective responses were obtained; and/or (3) measured in-situ $L_{\text{A,eq}}$ values above $77\text{\,}\mathrm{dB}$ in order to ensure safe listening levels [57] for the participants.

For each half of the binaural recordings that was not discarded, a $0\text{\,}\mathrm{\SIUnitSymbolDegree}$-azimuth, $0\text{\,}\mathrm{\SIUnitSymbolDegree}$-elevation field of view (FoV) was cropped out of their corresponding $360\text{\,}\mathrm{\SIUnitSymbolDegree}$-videos from the USotW database to form the audio-visual stimulus corresponding to that urban soundscape.

For the test set, six urban soundscapes were recorded in locations within Nanyang Technological University (NTU), Singapore. The recordings were made in a similar manner as those in the USotW database and were made using equipment in accordance with the SSID Protocol [53]. Post-processing of the test set recordings for use as part of the stimuli for the ARAUS dataset was done in the same manner as that for the five-fold cross-validation set.

In total, this formed a base of $234$ urban soundscape recordings for the five-fold cross-validation set of the ARAUS dataset, and an additional base of six soundscapes (not overlapping with the other $234$) for the independent test set. Further details on the exact recordings used can be found in Appendix A.

The masker recordings for both the five-fold cross-validation set and the independent test set were derived from source tracks found in the public databases Freesound [32] and Xeno-canto [58]. Both databases host tracks with Creative Commons licenses, with Xeno-canto being a repository of bird calls and Freesound being a more general repository of sound samples and recordings.

The source tracks from both databases that we determined to be relevant to the ARAUS dataset fell into one of the following classes: bird, construction, traffic, water, and wind. Water [59, 60, 61], bird [54, 61, 62], and wind [63] sounds have previously been investigated in soundscape studies as natural-sound maskers. On the other hand, sounds from traffic and construction are ubiquitous noise sources in urban environments [64], and are commonly investigated in soundscape literature [65, 66, 67]. Therefore, the selection of maskers covers a variety of urban sounds investigated in the soundscape literature for the ARAUS dataset.

The source tracks for maskers corresponding to the “bird” class were first obtained by randomly picking a selection of high-quality tracks of birds on Xeno-canto. Each source track corresponded to bird(s) from a single species, as labeled on Xeno-canto. Additional tracks for the “bird” class and all other classes were obtained via the corresponding search term on Freesound, and picking a selection of “high-quality” tracks containing $30$-second sections of sound that corresponded only to that particular masker class, as determined by manual listening. However, the exact number of sources of a single masker class present in a given track was variable, so for instance, a given track of the “bird” class could contain vocalizations from one, two, or more birds, but all tracks of the “bird” class contain only bird vocalizations. Further details on the source tracks used to create the maskers are given in Appendix A.

Each source track was then processed individually to create 30-second single-channel masker recordings. Single-channel recordings of maskers were used because [68] previously found single-channel recordings to be sufficient to replicate the perceived affective quality of soundscapes, which the ARAUS dataset aims to collect responses for. For source tracks that were originally multi-channel, only the first channel was used for consistency. Source tracks originally longer than $30\text{\,}\mathrm{s}$ were trimmed to $30\text{\,}\mathrm{s}$, while those originally shorter than $30\text{\,}\mathrm{s}$ were either padded with silence or looped. Finally, noise reduction via spectral gating and high-pass filtering was performed for source tracks in the “bird” class to reduce ambient and/or microphone noise in the track. All pre-processing was done manually using Audacity (v2.3.2).

In total, this formed a set of $280$ masker candidates ($56$ per fold) that were used to generate the stimuli for the five-fold cross-validation set, and a set of seven maskers for the independent test set. The breakdown of the number of masker recordings by class was $80$ bird, $40$ construction, $40$ traffic, $80$ water, and $40$ wind for the cross-validation set; and $2$ bird, $1$ construction, $1$ traffic, $1$ water, $2$ wind for the independent test set.

After preparing the urban soundscape recordings and maskers in Sections 3.1 and 3.2, the tracks were assigned into the five folds of the cross-validation set such that the distributions of psychoacoustic properties of the urban soundscapes and maskers were similar across the five folds. Since psychoacoustic indicators of a given soundscape have non-trivial and non-spurious correlations with corresponding perceptual indicators [69], taking psychoacoustic indicators into account was necessary to minimize distributional shifts across the folds.

The assignment procedure consisted of the following steps carried out for the urban soundscape recordings, and independently each class of masker tracks.

Each stimulus in the ARAUS dataset is an augmented soundscape to be presented as a $30$-second audio-visual stimulus to a human participant, and the procedure used to generate each stimulus is shown in Fig. 2.

The audio for the augmented soundscapes was made by combining the $30$-second binaural recordings of urban soundscapes in Section 3.1 with the $30$-second single-channel recordings of maskers obtained in Section 3.2 at various gain levels, via element-wise addition of their respective gain-adjusted time-domain signals. For the purposes of stimulus generation, we also included silence in the set of possible maskers, since the addition of silence simply replicates the condition where no masker is added.

For the five-fold cross-validation set, the urban soundscapes and maskers were chosen randomly from the same fold for combination to prevent data leakage and provide a sample of all possible combinations. Since the fold allocation procedure described in Section 3.3 ensured that the sets of urban soundscapes and maskers used for each fold of the cross-validation set were disjoint, the augmented soundscapes generated from them were disjoint between each fold of the cross-validation set as well. On the other hand, for the independent test set, the urban soundscapes and maskers were exhaustively combined to cover all $48$ possible combinations.

Before combining the urban soundscape and masker recordings for the five-fold cross-validation set, the urban soundscape recordings were calibrated to the in-situ $L_{\text{A,eq}}$ levels measured at the time of recording, and the maskers were calibrated to specific soundscape-to-masker ratios (SMR) with respect to the urban soundscape, chosen randomly from the set $\{$$-6$, $-3$, $0$, $+3$, $+6$$\}$ $\text{\,}\mathrm{dB}$. For example, if the urban soundscape recording had an in-situ $L_{\text{A,eq}}$ of $65\text{\,}\mathrm{dB}$ and an SMR of $+3\text{\,}\mathrm{dB}$ was randomly chosen, then the masker would be calibrated to an $L_{\text{A,eq}}$ of $62\text{\,}\mathrm{dB}$. This gave a total of $65\,520$ possible augmented soundscapes from which we sampled for the ARAUS dataset. All calibration was done via the automated method described by [80]. On the other hand, for the independent test set, a fixed SMR of $0\text{\,}\mathrm{dB}$ was used for all combinations. This was to limit the number of stimuli presented to each participant in the test set and the effect of listener fatigue, since the participants assigned to the test set were required to rate all combinations exhaustively, as explained in Section 3.7.

Lastly, the single-channel recordings were added to each channel of the binaural tracks, in a manner similar to that in [81] with a fixed stereo panning coefficient of 0.5. This audio was then overlaid onto the $0\text{\,}\mathrm{\SIUnitSymbolDegree}$-azimuth, $0\text{\,}\mathrm{\SIUnitSymbolDegree}$-elevation field of view cropped from the $360\text{\,}\mathrm{\SIUnitSymbolDegree}$ video recordings from the USotW database taken at the same time as the binaural urban soundscape recordings to form the audio-visual stimuli.

Prior ethical approval was obtained from the Institutional Review Board, NTU (Ref. IRB 2020-08-035) before participant recruitment and response collection. Participants were recruited via online messaging channels, posters, and emails.

In total, $642$ unique participants were recruited to provide their responses for the ARAUS dataset, of which $37$ ($5.76\text{\,}\mathrm{\char 37\relax}$) had their responses rejected. Hence, responses from only $605$ participants were included in the final dataset. We rejected responses from participants who (1) failed a hearing test ($19$ participants, $2.96\text{\,}\mathrm{\char 37\relax}$); (2) failed more than three out of seven consistency checks described in Section 3.9 ($17$ participants, $2.65\text{\,}\mathrm{\char 37\relax}$); or (3) provided the same responses to any item in the Affective Response Questionnaire (ARQ) described in Section 3.7 for all stimuli they were presented, thereby providing no useful information to the dataset ($3$ participants, $0.47\text{\,}\mathrm{\char 37\relax}$). Two of the rejected participants failed both the hearing test and more than three out of seven consistency checks.

Participants aged under $30$ were considered to have failed the hearing test if they had a mean threshold of hearing above $20\text{\,}\mathrm{dB}$ and participants aged $30$ and above were considered to have failed if they had a mean threshold of hearing above $30\text{\,}\mathrm{dB}$ via pure-tone audiometry using the uHear application on a mobile phone (Apple iPhone 4S) and earbuds (Apple EarPods). The tested frequencies were $0.5$, $1$, $2$, $4$, and $6\text{\,}\mathrm{kHz}$. These were within the standard ranges used for screening in pure tone audiometry [82] and previous soundscape research [83]. The higher threshold of $30\text{\,}\mathrm{dB}$ was applied to participants aged $30$ and above to balance the risk of age bias (since age is highly correlated with hearing ability [84]) against the need to ensure that hearing loss did not interfere with the perception of the augmented soundscapes in the ARAUS dataset.

The participants were each assigned a fold such that each fold of the five-fold cross-validation set had responses from $120$ participants. The independent test set had responses from $5$ participants. The age and gender distributions of the participants are shown in Table 3, and further information on participant demographics can be found in Appendix C. Due to the test sites for the data collection process being located within university campuses (as shown in Fig. 3), a majority of the 605 participants whose responses were included in the ARAUS dataset were students ($443$ participants, $73.2\text{\,}\mathrm{\char 37\relax}$) who were in the process of obtaining their bachelor’s degree ($380$ participants, $62.8\text{\,}\mathrm{\char 37\relax}$). Participants were also relatively young (mean age $26.7$ years, standard deviation $10.0$ years) compared to those in the ISD (mean age $33.8$ years, standard deviation $14.57$ years) [85], but slightly older compared to those in the IADS-2 (college students) [46] and IADS-E (mean age 21.32 years, standard deviation 2.38 years) [48] datasets.

Participants were presented with all audio-visual stimuli via closed-back headphones (Beyerdynamic Custom One Pro) powered by an external sound card (SoundBlaster E5). The video was presented via a $23$-inch monitor (Philips 236E SoftBlue) and measured $21.5\text{\,}\mathrm{cm}$ by $12\text{\,}\mathrm{cm}$ on the screen. Participants sat facing about $1\text{\,}\mathrm{m}$ from the monitor.

Due to the large number of participants involved in the experiment, participants listened to the stimuli in one of four quiet rooms, three located in NTU and one in the Singapore University of Technology and Design (SUTD). Photos of each of the four quiet rooms are shown in Fig. 3.

For each audio-visual stimulus, participants were instructed to perform their evaluations given the following prompt:

Imagine that you are standing at the location shown in the video, listening to the sound environment playing through the headphones.

After completely experiencing the stimulus at least once, participants were presented with a set of $9$ questions from the Method A questionnaire in ISO/TS 12913-2:2018 [20], which we refer to as the “Affective Response Questionnaire” (ARQ). The first $8$ questions were related to the perceived affective quality of the sound they heard over the headphones:

To what extent do you agree or disagree that the present surrounding sound environment is {pleasant, eventful, chaotic, vibrant, uneventful, calm, annoying, monotonous}?

The last question was related to the appropriateness of the location depicted in the video they saw on the monitor with respect to the sound they heard over the headphones:

To what extent is the present surrounding sound environment appropriate to the pleasant place?

Participants responded to all questions on a five-point Likert scale via a computerized graphical user interface (GUI) depicted in Fig. 4, and we coded their responses to values in $\{1,2,3,4,5\}$ to match the scale.

For the cross-validation set, participants responded to the ARQ for $42$ unique, randomly-selected stimuli from the fold that they were assigned to. For the independent test set, all participants responded to the ARQ for the same 48 exhaustively-generated stimuli.

In addition to the “main” $42$ ($48$) stimuli shown to each participant in the cross-validation (test) set, we presented each participant with three auxiliary stimuli. Participants were also required to provide responses to the ARQ for these stimuli, but the responses are not part of the ARAUS dataset, because these stimuli were the same for every participant, and did not serve the same purpose as the main stimuli in the ARAUS dataset. These stimuli were namely:

1. (1)

   A “pre-experiment” stimulus, which was shown as the first stimulus before presenting the main stimuli for the cross-validation/test set.

2. (2)

   An “attention” stimulus, which was identical to the “pre-experiment” stimulus and shown in between two randomly selected “main” stimuli for the cross-validation/test set. For this stimulus, the GUI had special instructions for the participant to choose specific options for the ARQ before they could proceed.

3. (3)

   A “post-experiment” stimulus, which was identical to the “pre-experiment” stimulus and shown as the last stimulus after presenting the main stimuli for the cross-validation/test set. ARQ responses to this stimulus were used for internal consistency checks, as described in Section 3.9.

Since the ISO 12913-1:2014 definition of “soundscape” mandates that acoustic environments must be perceived in context [86], information specific to the listeners who participated in the study was also necessary to understand the context behind listener perceptions of the augmented soundscapes presented as stimuli. Hence, we administered another questionnaire, which we dub the “Participant Information Questionnaire” (PIQ), to each participant after they had completed the ARQs for the stimuli shown to them. The PIQ consisted of items related to basic demographic information and standard psychological questionnaires.

Items related to basic demographic information consisted of age, gender, spoken languages, citizenship status, education status, occupational status, dwelling type, ethnicity, and length of residence of the participant.

The psychological questionnaires administered were (1) a shortened, $10$-item version of the Weinstein Noise Sensitivity Scale (WNSS-10) [87]; (2) a shortened, $10$-item version of the Perceived Stress Scale (PSS-10) [88]; (3) the WHO-5 Well-Being Index [89]; and (4) the Positive and Negative Affect Schedule (PANAS) [90].

The relevance of the questionnaires is evident in the fact that noise sensitivity [91] and stress level [92] play a significant role in the affective perception of soundscapes, the WHO-5 Well-Being Index is used in the Soundscape Indices Protocol (SSID) [53], and [93] previously used PANAS as a measure of participants’ mood in a study of the emotional salience of sounds in soundscapes. The PSS-10 has been validated by the same authors of the original 14-item questionnaire [94], but the particular version (WNSS-10) of the Weinstein Noise Sensitivity Scale used for the ARAUS dataset has not previously been validated in the literature. Nonetheless, the internal reliability of WNSS-10 is affirmed based on the results of the analysis obtained in Section 4.2.

Full details of the PIQ with individual questions, options, and response coding for each section of the questionnaire can be found in Appendix B.

In order to ensure a baseline level of data quality in the responses to the ARQ, seven consistency checks were performed on each participant’s responses. The consistency checks were designed as single-value metrics, and a participant was considered to have failed a consistency check if the corresponding value of the metric was at least $1$. If a participant failed more than three out of seven consistency checks, their responses were dropped from the dataset.

To describe the single-value metrics, we first define $r_{\text{pl}},r_{\text{ev}},r_{\text{ch}},r_{\text{vi}},r_{\text{ue}},r_{\text{ca}},r_{\text{an}},r_{\text{mo}},r_{\text{ap}}\in\{1,2,3,4,5\}$ as the responses to the ARQ in Section 3.7 regarding the extent to which the sound environment in a given stimulus was respectively pleasant, eventful, chaotic, vibrant, uneventful, calm, annoying, monotonous, and appropriate.

The seven consistency checks consist of three types of checks. The first check measures the mean absolute difference (MAD) between ARQ responses on the “pre-experiment” and “post-experiment” stimuli. Since the “pre-experiment” and “post-experiment” stimuli were identical, a perfectly consistent participant would have provided the same responses to both presentations.

Since the affective descriptors “pleasant” and “annoying” are on opposite axes of the ISO/TS 12913-3:2019 circumplex model of soundscape perception [11], a perfectly consistent participant would have provided the same response for “pleasant” and “annoying” if the Likert coding of either were to be reversed. The same considerations apply to the other three pairs of opposite attributes on the circumplex model. Therefore, the next four checks consider the MADs between the four pairs of $r_{p}$ and $(6-r_{q})$, where $(p,q)$ is a pair of opposite descriptors.

Lastly, the mean squared error (MSE) between $r_{\text{pl}}$ and $(3+2P)$, and that between $r_{\text{ev}}$ and $(3+2E)$ across all stimuli presented was computed, where

respectively are the normalized values of “ISO Pleasantness” and “ISO Eventfulness” as suggested in [95], and $k=8+\sqrt{32}$ is a normalization constant such that $P,E\in[-1,1]$. Since the affective descriptor “pleasant” theoretically parallels the principal axis of “Pleasantness” in the ISO/TS 12913-3:2019 circumplex model of soundscape perception, a perfectly consistent participant would have provided responses matching the magnitude in both directions. The rescaling of $P$ as $(3+2P)$ was necessary to match the range of values of $r_{\text{pl}}$ for a valid comparison. Similar considerations apply for the affective descriptor “eventful” to the principal axis “Eventfulness”.

To investigate the effect each masker had on the ISO Pleasantness of the urban soundscapes it was augmented to, we first used the ARQ responses to compute the ISO Pleasantness values for all the stimuli in the ARAUS dataset using Equation (1). The difference in ISO Pleasantness for each base urban soundscape (i.e., augmented with silence) and the same urban soundscape augmented with each masker was computed. These differences were averaged across all soundscapes for which the same masker, regardless of the SMR, was presented to obtain the mean change in ISO Pleasantness effected by each masker across different soundscapes. This allowed us to determine which maskers were optimal for augmentation, at least on average in a naive sense across the urban soundscapes in the ARAUS dataset.

Figure 5 shows the mean change in ISO Pleasantness value as a function of masker used to augment soundscape, aggregated over soundscapes and SMRs used. Out of the $287$ maskers in the ARAUS dataset, mean positive changes in ISO Pleasantness were only observed in maskers belonging to the bird and water classes. However, not all of the bird and water maskers showed mean positive changes, corroborating the findings by [92] that not all natural sounds are necessarily perceived as pleasant. Nevertheless, augmentation with $64$ ($78.0\text{\,}\mathrm{\char 37\relax}$) of the bird maskers and $16$ ($19.5\text{\,}\mathrm{\char 37\relax}$) of the water maskers, each out of $82$, resulted in effective mean positive changes in ISO Pleasantness, which supports findings in [14, 54, 70] where specific bird and water sounds were found to have improved the perceived pleasantness of urban soundscapes.

In contrast, mean negative changes in ISO Pleasantness were observed for all wind, traffic, and construction maskers in the ARAUS dataset. While the decrease in ISO Pleasantness due to the addition of traffic [96] and construction [97] maskers is expected, the decrease in ISO Pleasantness for all wind sounds used as maskers was contrary to the results in narrative interviews reported by [92] that people tended to perceive wind rustling through trees as pleasant. This could be due to the range of SMRs used for the ARAUS dataset, which ranged only from $-6$ to $+6\text{\,}\mathrm{dB}$. The mean in-situ $L_{A,eq}$ of the tracks in the USotW database used for the ARAUS dataset was about $65\text{\,}\mathrm{dB}$, but natural wind sounds in real-life environments tend to have a mean $L_{A,eq}$ of about $55\text{\,}\mathrm{dB}$ or less [98], which means that an SMR of $+10\text{\,}\mathrm{dB}$ or higher may have been more appropriate to achieve an increase in perceived pleasantness for wind maskers instead. At the SMRs used for the ARAUS dataset, the wind maskers may have been added at overly high levels, rendering them to be perceived similarly to the traffic maskers due to their similar spectral characteristics and potentially contributing to the decrease in ISO Pleasantness. Additionally, the laboratory-based nature of the data collection process caused the wind maskers to be heard without perceiving natural movements of the air that would be present in an in-situ study, which could have resulted in an artificial or unpleasant situation for the participants.

To investigate how well the ARAUS dataset stimuli and the base urban soundscapes from the USotW database spanned the perceptual space generated by the “Pleasantness–Eventfulness” axes, we first computed the normalized values of $P$ and $E$ according to Equations (1) and (2) for each individual ARQ response in the ARAUS dataset. Then, we plotted a heat map and scatter plot of the responses on the $P$–$E$ axes, as shown in Fig. 6. We can see that both the ARAUS dataset stimuli and the USotW soundscapes covered the positive, neutral, and negative regions of both the ISO Pleasantness and the ISO Eventfulness axes, thereby validating their use in analysis related to the ISO 12913 standard.

Since items in the PIQ such as the participant’s age [99] and education status [100] can affect the affective perception of soundscapes, the participants in each fold of the cross-validation set of the ARAUS dataset should ideally be drawn from distributions with similar demographics. This was not enforced during the participant recruitment and fold allocation process, thus a post hoc analysis of the PIQ responses was performed to assess the demographic distributions. The post hoc analysis was performed via standard statistical tests for equality of distributions.

For items in the PIQ coded as categorical variables, $\chi^{2}$-tests were conducted between the responses obtained in each fold of the cross-validation set (treated as observed frequencies) and those obtained in the entire cross-validation set (treated as expected frequencies). No significant differences were observed, at $5\text{\,}\mathrm{\char 37\relax}$ significance levels, in the distribution of all categorical variables between folds, with the lowest $p$-value being $0.1070$ for the participants’ gender.

For items in the PIQ coded as continuous variables, Kruskal-Wallis tests were conducted by treating each fold as an independent group. The Kruskal-Wallis tests showed no significant differences at $5\text{\,}\mathrm{\char 37\relax}$ significance levels in the distribution of all continuous variables between folds, with the lowest $p$-value being $0.3090$ for the extent to which participants were annoyed by noise over the past $12$ months.

Together, these results indicate that the distribution of participants in a given fold did not significantly differ from the participants in any other fold, which reinforces the validity of using the five-fold cross-validation set of the ARAUS dataset to generate models for populations sharing similar characteristics to that of the entirety of the cross-validation set. Full results of the statistical tests and explicit distributions of responses by fold and PIQ item are detailed in Appendix C.

For the psychological questionnaires used in the PIQ (WNSS-10, PSS-10, WHO-5, and PANAS), Cronbach’s $\alpha$ [101] and McDonald’s $\omega$ [102] were computed as standard internal reliability coefficients to independently verify if their items in aggregate were indeed measuring the same construct. This served as a validation study for WNSS-10 and a replication study for the other questionnaires.

The internal reliability study results are shown in Table IV. All values of the internal reliability coefficients are above 0.8, which can be considered “high” given the number of items in each questionnaire [103]. Therefore, all items in each questionnaire indeed measured the same construct. In particular, the Cronbach’s $\alpha$ value for of $0.835$ for WNSS-10 parallels similar validation studies for different iterations of the questionnaires of the original $21$-item version [104], thereby confirming the reliability of the previously unvalidated version used for the ARAUS dataset.

In a similar manner to the PIQ, Kruskal-Wallis tests for each of the seven single-value consistency metrics were performed by treating the ARQ responses in each fold as independent groups. Full results are presented in Appendix C.

No significant differences were observed, at $5\text{\,}\mathrm{\char 37\relax}$ significance levels, in the distribution of all continuous variables between folds, with the lowest $p$-value being $0.2545$ for the MAD between “vibrant” and reversed “monotonous” ratings. This indicates that the distribution of response consistency in a given fold did not significantly differ from the responses in any other fold, reinforcing the validity of using the five-fold cross-validation set of the ARAUS dataset to generate generalizable and unbiased models, at least from the perspective of consistency with the ISO 12913-3:2019 circumplex model of soundscape perception.

The label means for the normalized ISO Pleasantness $P$ were $0.0208$, $0.0317$, $0.0307$, $0.0281$, and $0.0225$ when folds $1$, $2$, $3$, $4$, and $5$ were respectively left out as the validation fold for the derivation of the dummy models. Hence, these were also the exact predictions given by all dummy models regardless of the soundscape presented. Considering that $P\in[-1,1]$, the difference in label means by fold is also insubstantial, since they are within $0.5\text{\,}\mathrm{\char 37\relax}$ of the full range. In fact, the distributions of the training set labels themselves have no significant difference by fold, as shown in Fig. 8.

After training the elastic net models, the weights for all but three input features were set to zero regardless of the fold used as the validation set. Taking the mean of weights for the regression models across the 5-fold cross-validation set, this gave the general regression model

where $N_{\text{max}}$ is the maximum loudness (in sone), $M_{f}$ is the power (in $\text{\,}\mathrm{dB}$) at the one-third octave band with center frequency $f$, the prediction for the normalized ISO Pleasantness value $P$ is denoted as $\widehat{P}$, and the regression coefficients are not normalized.

This implies that of the $264$ acoustic and psychoacoustic parameters used as input features to train the regression models, the features that most significantly impacted the ISO Pleasantness were $N_{\text{max}}$, $M_{\text{$10\text{\,}\mathrm{kHz}$}}$ and $M_{\text{$12.5\text{\,}\mathrm{kHz}$}}$, at least based on the responses in the ARAUS dataset. Incidentally, for the stimuli in the ARAUS dataset, we observed that the ranges of these features were $N_{\text{max}}\in[8.36,94.1]$, $M_{\text{$10\text{\,}\mathrm{kHz}$}}\in[29.62,85.15]$ and $M_{\text{$12.5\text{\,}\mathrm{kHz}$}}\in[32.51,78.67]$, which meant that the predictions had a range of $\widehat{P}\in[-0.6121,0.1680]$. The negative coefficients of the features in Equation (3) also suggest that increasing the peak loudness and high-frequency components of an augmented soundscape correlate with a decrease in its perceived pleasantness, corroborating the respective findings by [109] and [110] regarding general acoustic environments.

Lastly, Table V compares the performance of the four baseline models and the dummy model with respect to the MSEs achieved across their training, validation, and test sets in the $5$-fold cross-validation scheme used to train them. All baseline models performed better than the dummy model in the training, validation, and test sets, which suggests that the features trained in all the baseline models to make ISO Pleasantness predictions were indeed meaningful.

In addition, the CNN and PPAP models performed significantly better than the elastic net models, with the lowest mean train, validation, and test set MSE of $0.1086$, $0.1212$, and $0.0838$ observed with the PPAP performing feature-domain augmentation [106], the CNN [41], and the PPAP performing feature-domain augmentation [106], respectively. This is expected due to the vast difference in the numbers of parameters used by the CNN and PPAP models (142–515K) for prediction, in comparison to the regression models ($4$ parameters) and dummy model. Nevertheless, none of the models were overfitted to the dataset, as can be seen from the relatively similar values for training and validation set MSEs across all models, which demonstrates the versatility of the ARAUS dataset for use in training generalizable high-parameter models.

Notably, the test set MSE values are also smaller than that of the validation set, which could be due to the relative size of the test set ($48$ samples) as compared to the validation set ($5040$ samples) making the test set “easier” from the perspective of the prediction models than their respective validation sets.