2.1 Judges and Voters

Between 10 April 2020 and 10 May 2020, voters were able to rate the AI Song Contest entries on either of two websites, one in Dutch and one in English. When multiple votes for the same song arrived from the same IP address, only a single vote, chosen at random, was kept. No demographic information was asked of or recorded for the voters, but based on IP addresses, the contest sites attracted 3826 voters across 70 different countries. Of these 70 countries, however, just four accounted for more than half of the voters: The Netherlands (22%), Belgium (16%), Australia (14%), and France (11%).

Alongside the public vote, an international panel of three researchers in music and artificial intelligence served as a jury for the contest. In addition to the songs themselves, each participating team provided the jury with a ‘process document’ explaining how they had created their song and how the interaction between humans and algorithms had worked. After listening to all the entries and reading all of the process documents, the jury met to discuss and evaluate them on a separate list of criteria from the public voters.

2.2 Evaluation Criteria

The public voting sites asked voters to evaluate the entries on four criteria, using scales of 0 to 3: its originality, the quality of the song itself, its ‘Eurovision-ness’, and its lyrics. Although this system was quite different than the ranked-choice system of the Eurovision Song Contest, it shared one common detail: the maximum possible total score across all criteria was 12 points. A complete spreadsheet of the public votes is available as Supplementary File 1.

The judges on the jury evaluated the entries according to four other criteria: effective and creative use of AI (scale of 0 to 6), expansion of creativity (scale of 0 to 2), furthering understanding (scale of 0 to 2), and diversity and collaboration (scale of 0 to 2). Again, the maximum possible total score across all criteria was 12 points. Although the judges kindly shared a spreadsheet of their evaluations with us for the purposes of this paper, in order to protect the anonymity of their individual opinions and the integrity of the contest, we cannot make them public.

The final ranking was determined by the total of the average voter score (summed across all criteria) and the average jury score (summed across all criteria and rounded to the nearest integer) for each entry. Overall, the evaluation scheme was in line with recommendations for evaluating computational creativity (e.g., SPECS, Jordanous, 2012), incorporating aspects of the process as well as the product, multiple recognised dimensions of creativity, and a combination of expert and non-expert evaluators. Figure 1 shows the development of the average voter scores during the voting period and a comparison of the voters’ final scores against the jury’s.

2.3 Distribution of Votes and the Groupie Effect

In total, there were 12,416 votes (defined here as one voter sending a completed evaluation rubric for one song). Only 11 percent of voters voted for every song, but these voters were responsible for 43 percent of the total number of votes. At the other extreme, 67 percent of voters voted for one song only, but due to this group’s low vote count, they were responsible for only 21 percent of the total number of votes. The votes from one-time voters were distributed quite unevenly over the teams (see the heatmap in Figure 2): while 45 percent of votes for ‘Beautiful the World’ came from one-time voters, the same was true of only 2 percent of voters for ‘Painful Words’. One-time voters were also substantially more likely to award perfect scores than other voters: whereas perfect scores comprised only 8 percent of votes from voters who voted for two or more songs, one-time voters gave their teams a perfect score 69 percent of the time. We dub this phenomenon the groupie effect.

2.4 Rasch Models

We analysed the data using the Rasch approach to psychometric measurement (Rasch, 1960; Wright and Mok, 2004), specifically the Rasch rating-scale model and the Rasch partial-credit model (Wright and Masters, 1982). These are models for passing each step of a rating scale, conditional on having passed all previous steps. They are closely related to logistic regression, and with the right parameterisation, it is possible to fit these and most other Rasch-type models using any statistical software that supports hierarchical log-linear modelling.

Consider a rating scale with integer scores ranging from 0 to K. Given a quality parameter θ_n for a song n, an overall difficulty parameter δ_i for a rating criterion i, and a set of thresholds τ_k for the rating scale, k ∈{1, …, K}, the Rasch rating-scale model for the probability of passing the threshold to x_ni is

The partial-credit model is a variant that allows an interaction between criteria and the rating thresholds (i.e., the perceptual distance between scale steps may differ across criteria). Instead of a common set of thresholds τ_k, it combines criterion difficulties and thresholds into a matrix of parameters δ_ik:

In contests like the AI Song Contest, wherein multiple judges and voters rate the same songs on a common set of criteria, these models can be extended to so-called many-facet Rasch models by adding severity parameters λ_j to reflect the harshness of judge or voter j when applying the rating criteria (Linacre, 1989). The rating-scale model becomes

and the partial-credit model:

Marginalising over all possible ratings yields the usual form for presenting these models:

for the rating-scale model and for partial credit:

The most notable feature of Rasch-type models is what they do not include: namely, any kind of interaction between the quality of the songs, difficulty of the rating criteria, and severity of the judges or voters. Unlike item–response theory in general, the Rasch approach excludes these interactions from consideration on principle, even where they would improve model fit. This viewpoint can be controversial, but the argument for it is straightforward when the goal is not just a model for predictive inference but also a robust measurement model that can generalise to new circumstances (Andrich, 2004). Under Rasch models, the total raw scores for each team, criterion, and judge or voter are sufficient statistics for the corresponding parameters in the model; moreover, in the case of complete data, the relationship between raw scores and parameter values is monotonic. Crucially, in the case of incomplete data or, especially relevant to the task of evaluating a song contest, in the case of a new set of songs, a different set of rating criteria, or a different group of judges and voters, the underlying parameter values in Rasch models remain unchanged. This property, known as specific objectivity (Rasch, 1977), is so essential to measurement from the Rasch perspective that it is preferable to remove outlying songs, criteria or judges before introducing an interaction that would destroy specific objectivity.

2.5 Reliability

Similar to the intra-class correlation coefficient (ICC) and other classical measures of reliability (e.g., Lord and Novick, 1968), reliability in Rasch models for a parameter of interest φ is defined as

where ${\sigma }_{\epsilon }^2$M10 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[\sigma _\varepsilon ^2\] \end{document} is the variance of measurement error for the parameter of interest and ${\sigma }_{\theta }^2$M11 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[\sigma _\theta ^2\] \end{document} is the variance of the (estimated) parameter of interest across the population (Wright and Masters, 1982). The maximum possible value of ρ is 1, and intuitively, it represents the proportion of variance in parameter estimates that represent true differences in quality, difficulty, or severity (as opposed to measurement error). Conventionally, after Nunnally (1978), ρ ≥ 0.7 is considered good and ρ ≥ 0.9 is considered excellent.

Because of specific objectivity, reliabilities in Rasch models can be computed independently for the estimates of song quality, criterion difficulty, and voter or judge severity. In many-facet models, Rasch reliability is also robust to cases of perfect separation where classical measures of inter-rater reliability can fail catastrophically (Bond et al., 2020).

2.6 Groupies and Three Inflation

As mentioned earlier, some AI Song Contest entries attracted large numbers of voters who voted only for one entry and gave perfect scores. In principle, many-facet Rasch models can handle this situation by assigning very low severities λ_j to the one-time, perfect-score voters; indeed, with no prior or other sort of regularisation, the maximum-likelihood estimate of the severity of such a voter would be –∞. But one could also imagine a distinct dimension that is responsible for this pattern of high scores in one-time voters: that while some perfect scores are ‘true’ perfect scores arising from a very positive assessment of a song’s merits, others arise from a simple popularity contest, whereby some voters (the groupies) simply logged in to give a perfect score to their friends. Alternatively, one could imagine that one-time voters indeed did listen carefully to one or more entries, but then simply gave a perfect score to their favourite rather than going to the trouble of entering comparative ratings.

In either scenario, one would expect to see a larger number of threes in the rating data (the highest possible score on the rating criteria) than one would under a pure Rasch model. These scenarios are similar to those where researchers might use zero-inflated models for count data (e.g., the number of tracks streamed by a listener during each hour of the day). Zero-inflated models consider counts to come from a Poisson distribution ‘inflated’ with extra zero counts (Lambert, 1992). Specifically, the zero-inflated Poisson model assumes that counts are drawn not from a pure Poisson distribution but rather from a mixture distribution, with a probability γ of a count arising from a process that generates only zeros (e.g., the probability that the user is not listening to music during a particular hour) and probability 1–γ that it arises from the Poisson distribution of interest. We use the same principle to derive a three-inflated Rasch model: instead of the pure Rasch models in Equations (5) and (6), we consider mixture models in which there is a probability γ that a rating is a three regardless of the underlying song quality – either because of pure popularity or because a voter chose to rank only their favourite song – and a probability 1–γ that the rating is sampled from the distribution of a Rasch model.

The structure of the AI Song Contest data introduces an extra complication to this model. Our primary motivation for considering a three-inflated model is that the extra threes are not identically distributed, and as such, there should be multiple mixing parameters for different groups within the data. At first glance, it would seem that one would want to assign γ_j individually to each voter, on the idea that some voters, especially one-time voters, assigned ratings without specific regard to song quality whereas others used the scale more carefully. Even with regularisation, however, such a model introduces so many regions of perfect or near-perfect separability that it is difficult to estimate severities accurately. A more practical alternative is to consider the mixing probabilities γ_n to be parameters of the songs, capturing the idea that some entries, regardless of their quality, were more likely to attract groupies. The three-inflated many-facet partial-credit model is thus:

The rating-scale and single-facet variants of the three-inflated model are formed similarly.

3.1 Bayesian Hierarchical Model

In order to understand the voting data fully, not only are the parameters of the Rasch models of interest, but also their distribution. As such, we chose a Bayesian approach for model estimation. We coded the models as hierarchical models in the Stan language for Bayesian modelling, version 2.26 (Stan Development Team, 2021). To simplify the fit and facilitate comparisons across the jury criteria, we split the judge’s ratings for the use of AI, originally on a scale of 0 to 6, into three ratings on a scale of 0 to 2 (e.g., a rating of 5 would become three ratings of 1, 2, and 2, respectively). The jury contained only three judges, against 3862 voters, and so we weighted the jury observations and the voter observations by their inverse frequency when computing the log likelihood, so that the jury as a whole and the voters as a whole would make equal contributions to the posterior, as in the contest’s official scoring system.

The prior and hyper-prior distributions for the model are summarised in Table 2. For consistency with the other parameters, we work with the γ_n parameters on a logit scale rather than as raw probabilities. For partial-credit models, we introduce an interaction parameter ζ_ik to the rating-scale model and model the rating-scale thresholds as δ_i + τ_k + ζ_ik instead of the more direct δ_ik formulation in Equations (4) and (6); this expanded parameterisation allows us to eliminate any covariance between parameters in the prior distribution. In order to ensure regularisation and partial pooling in the case of parameters with fewer data points (e.g., the severity parameters for groupies), we assume that all parameters are normally distributed with a distinct variance hyper-parameter per group. While most parameters are centred at 0, an intercept term must be assigned to one of them in the form of a non-zero mean; we choose the criterion difficulties in order to facilitate interpretation (see below). We give all hyper-priors a standard normal or half-normal distribution, a weakly informative choice that provides further regularisation to a difficult posterior geometry (cf. Lemoine, 2019).

For reporting purposes, we convert the parameters to a standard T-score scale (Seashore, 1955) by multiplying all parameters by 10/σ_θ and adding 50 to the θ_n and δ_i parameters. This transformation has several advantages for interpretation. It fixes the scale of the prior distribution on the most important parameters of interest, the song quality parameters θ_n, such that they can be interpreted as arising from a normal distribution with a mean of 50 and a standard deviation of 10.³ All other parameters can then be interpreted relative to this fixed scale. For example, if Voter A is 10 points more severe than Voter B, Voter A will judge a song as if it were one standard deviation lower in quality than Voter B would. By assigning both the intercept and the 50-point shift to the criterion difficulties δ_i, the rating thresholds also have a neat interpretation: if the standardised rating threshold k for some criterion is equal to the standardised quality score of some song, then if an average judge is debating between ratings of k and k–1, they will assign either rating with equal probability (i.e., a coin toss). If the criteria are well targeted for evaluating AI Song Contest entries, the standardised rating thresholds should fall within a similar range to the standardised song quality scores (roughly 20–80). The closer the thresholds are to the song qualities, the more statistical information each vote provides, and thus the fewer the votes necessary to achieve a desired level of reliability (Wright and Masters, 1982).

For computing reliabilities of individual parameters according to Equation (7), we take the numerator ${\sigma }_{\epsilon }^2$M12 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[\sigma _\varepsilon ^2\] \end{document} to be the squared MAD SD (median absolute deviation scaled to match the standard deviation of a normal distribution) of the posterior distribution of the parameter of interest and the denominator ${\sigma }_{\phi }^2$M13 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[\sigma _\varphi ^2\] \end{document} to be the squared median of the posterior distribution of the corresponding hyper-prior (e.g., σ_θ for song qualities or σ_λ for voter severities). When reporting reliabilities for a group of parameters (e.g., the overall reliability of song quality measurements), we take the median over the reliabilities of all parameters in the group.

The Stan code for these models, including the T-score transform, is available as Supplementary File 2.

3.2 Model Selection

We considered all of the Rasch model variants discussed above: rating scale and partial credit, single- and many-facet, with and without three inflation. We sampled these models with Stan’s default no-U-turn sampler. In order to avoid divergent transitions, we used a very conservative adapt_delta setting of 0.99 and increased the maximum tree depth to 25. We ran four parallel chains for 1000 iterations each, discarding the first 500 samples from each chain as warm-up. This procedure was sufficient for all $\widehat{R}$M14 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[\hat R\] \end{document} statistics to converge to less than 1.01 and left us with 2000 samples from the posterior distribution of each model to use for further analysis. As a baseline, we also sampled from an intercept-only model, with all parameters except the song parameters and μ_δ fixed to 0:

We compared the models using leave-one-song-out cross-validation, which we expect to generalise better than randomised or leave-one-vote-out cross-validation structures for the purposes of predicting the ratings for new songs in new competitions (Merkle et al., 2019). The posterior geometry of hierarchical models like ours can be challenging, however, for the Markov-chain Monte Carlo (MCMC) sampling techniques that are used in modern Bayesian inference (see Stan Development Team, 2021, §22.7, ‘Reparameterization’). On current hardware, it takes 12 to 24 hours per model or fold before MCMC sampling converges,⁴ which renders full cross-validation impractical. Instead, we used a popular approximate cross-validation technique for Bayesian analysis: Pareto-smoothed importance sampling (PSIS, Vehtari et al., 2017). Even this approximate technique requires the likelihood of each observation for each sample to be integrated over the prior distribution of song quality in that sample, which is itself computationally demanding. We thus made a further approximation by simply evaluating the likelihood of each observation for each sample as if the song had been of the overall mean quality (i.e., at θ_n = 0).⁵

Note that this approach heavily favours the intercept-only baseline: it will select only models that can consistently capture more variance from criterion difficulties, judge or voter severities, and three inflation than they can capture from song quality. This conservatism is inherent in our choice to prefer leave-one-song-out cross-validation, which demands more robust parameters than a leave-one-vote-out model that would only be trying to predict how a voter from the 2020 contest might have rated one of the songs that they skipped.

Table 3 summarises the results. For each model, we report the total number of parameters and the LOO-IC. The LOO-IC is asymptotically equivalent to the more familiar Akaike information criterion (AIC); smaller values are better. Only three models outperformed the baseline: from worst to best, the three-inflated many-facet rating-scale model, the three-inflated single-facet partial-credit model, and the many-facet partial-credit model without three inflation. Bearing in mind that these values are all approximations, the results do suggest that the many-facet models are an improvement over the single-facet models – or in other words, that judge and voter severities matter. The evidence for or against three inflation is more equivocal: while it clearly can help in some cases, being part of two out of three of the best models, in other cases, it seems to degrade predictive performance. In the case of our best-performing model, however, it is perhaps not so surprising that three inflation is less helpful. Partial-credit models can accommodate more three ratings than a strict rating-scale model, and likewise many-facet models can accommodate more threes than single-facet models. When both partial credit and extra facets are already available, the extra parameters in a three-inflated model may be less necessary.

On the basis of these results, we selected the many-facet partial credit model without three inflation, Equation (4), for further reporting here. But it should be noted that the second-best model, the three-inflated single-facet partial-credit model, requires only 2% as many parameters, and this ratio would become even starker as the number of voters increased. The number of parameters affects both computation time and the storage necessary to save samples, and so in a larger contest, the three-inflated model might be preferable.

3.3 Song Calibrations

Figure 3a presents our main result: T-scaled calibrations from the model for song quality. They correlate almost perfectly with the official results (r_s = .97), with only some slight differences in ordering in the bottom half. The quality of measurement in terms of Rasch reliability was excellent overall (ρ = .90), although the last-place entry, ‘Painful Words’, was somewhat more difficult than the others to place precisely (ρ = .75). At this level of reliability, one can distinguish three or four quality levels: one group including the top three or four entries (which are statistically indistinguishable), another between that group and the average standard score of 50, a below-average group, and possibly ‘Painful Words’ in a class by itself. New Piano, the team behind ‘Painful Words’, interpreted the AI component of the challenge much more strictly than the other teams, eschewing as much human involvement as possible.

3.4 Criterion Calibrations

The average criterion difficulty was 40 on the standard scale, 95% CI [24, 56].⁶ The standard deviation was 17.5, 95% CI [9.6, 32.9], composed of three subcomponents: the standard deviation of mean criterion difficulty, 4.7, 95% CI [0.2, 14.5]; the standard deviation of the mean threshold offsets, 13.8, 95% CI [6.2, 29.8]; and the standard deviation of the criterion–threshold interaction that characterises partial-credit models, 8.0, 95% CI [4.1, 14.7]. Figure 3b illustrates the posterior distribution for the voters’ criterion thresholds and Figure 3c the jury’s. Given the much larger amount of data, the calibrations for the voters’ criteria are more precise.

To achieve optimal reliability at minimal cost, one would want the rating thresholds to be centred exactly around 50, the centre of the songs’ quality distribution. Ours are centered one standard deviation lower, which has the consequence of degrading reliability slightly for the top-ranked entries. That is undesirable for a competition, where the most important reason to evaluate is to determine a winner, but even with this degradation, the reliability of measurement for the winner was still good (ρ = .83).

The larger problem revealed by these calibrations is that the voters had too many choices. As a general guideline, rating thresholds should be separated by between 1.4 and 5.0 logits (Linacre, 2002), which corresponds to a separation of roughly 20 to 60 points on our standard scale (the median of the posterior distribution for σ_θ was 0.77). All of the voters’ criteria fail to meet this guideline, and future competitions could consider reducing the voters’ rating scales to a simpler 0 to 2 range. The jury used a 0-to-2 range for their evaluation, and with the notable exception of the diversity criterion, which operated as a de facto binary variable, this shortened range worked better.

The last notable pattern in the criterion calibrations is that the jury as a whole was more sensitive to difference at the top of the quality distribution than the voters were. Juries are brought in to most competitions in hope that the contest will benefit from a higher level of expertise, and these data suggest that it did indeed help the AI Song Contest.

3.5 Voter and Judge Calibrations

Voters’ and judges’ severity of assessment varied widely: 19.8 points on the standard scale, 95% CI [12.7, 28.3]. The posterior distribution was bimodal, however, and the pattern can be understood best by examining how severity interacted with the contest entries. As noted earlier, the one-time voters were clustered around only a few entries. Figure 3d shows density estimates of the severities of the voters who voted for each of the entries. The groupies are visible as sharp peaks on the negative sides of the distributions: low severity means a higher chance of awarding a perfect score. Figure 3e is the same visualisation, but excluding all voters who gave exclusively threes or exclusively zeros (1805 of the 3826 voters). The groupie effect disappears, and the severity distributions for each entry become similar.

Re-fitting the model without the groupies leaves the song quality parameters essentially unchanged. This is to be expected: many-facet Rasch models will naturally discount groupies because their very low severities mean that even in the case of multiple votes, they provide very little information about underlying song quality. The issue is that raw scores, which usually track Rasch calibrations quite closely, become distorted in the presence of large numbers of groupies.

3.6 Posterior Predictive Checks

Having worked through the model’s calibrations, it is worth making an extra check to ensure that it is indeed making good predictions. Figure 4 shows the results of a graphical posterior predictive check. The histograms reflect the actual ratings collected for the AI Song Contest. Using the 2000 samples from our model’s posterior distribution, we generated 2000 random datasets of the same size and structure as the actual data. The black bars in the figure cover 95 percent of the range of these simulated data, and in a perfect model, they would always include the tops of the histogram bars.

Overall, the model seems to have captured the structure of the data well, but there are a few notable outliers. ‘I’ll Marry You Punk Come’ was much more appreciated by the jury than it was by the voters, and the model seems to have split the difference. ‘Traveller in Time’ had no lyrics at all, and as such, it scored unexpectedly poorly on the lyrics criterion. And despite the overall weak performance of Painful Words, voters appreciated its originality more than the model did.