PhyloLM: Inferring the Phylogeny of
Large Language Models and
Predicting their Performances in Benchmarks

In the current landscape, LLMs predominantly operate on an autoregressive basis, wherein they learn the conditional probability denoted as $LLM(t|c)$. Here, $LLM$ represents the probability learned by the language model, $t$ signifies a token, and $c$ denotes the context in which to sample token $t$. Transposing genetic methods to LLMs involves establishing analogies for the elements of the phylogenetic analysis, namely genes, alleles, and populations. Drawing a parallel with the notation for populations in the Nei genetic distance (see Equation 1) [41], $P_{1}(a|g)$, we propose that LLMs play the role of populations (i.e., the set of the individuals belonging to a given population); contexts (or "prompts") are aligned to genes (i.e., portions of DNA); finally, tokens align with alleles (i.e., variants in the DNA sequence).

To substantiate this analogy, consider that, in the realm of genetics, populations are conceptualized as probability distributions of DNA, represented by $P(a|g)$, where $a$ stands for specific alleles at gene locations. Gene-specific alleles are then considered to be probabilistically drawn from the abstract statistical construct that is the population, akin to context-specific tokens are probabilistically generated from Large Language Models, expressed as $LLM(t|c)$ ($t$ being a token likely to follow text $c$). The generated text can therefore be seen as a thread of DNA, comprised of tokens (alleles) sampled in contexts (genes) according to a probability distribution defined by the LLM.

To elucidate this crucial point, consider a tokenized text sequence: ’I’ ’_like’ ’_choco’ ’late’. This sequence can be analogous to a DNA thread represented as ’I_like_chocolate’. Breaking it down, the allele I corresponds to the gene $\epsilon$ (empty text), _like aligns with the gene I, _choco associates with the gene I_like, and late is linked to the gene I_like_choco. Now, consider another individual represented by ’I’ ’_prefer’ ’_ice’ ’_cream’. These two individuals share exactly two genes: $\epsilon$, for which they possess the same allele I, and the gene I, for which they have distinct alleles (_like and _prefer). They do not share any further genes, as their prefixes diverge beyond this point.

Implementing the phylogenetic algorithm in the context of Large Language Models involves the somehow arbitrary selection of specific genes¹¹1Which is not uncommon in genetics as well. Evolutionary studies are generally about a series of well-selected genes and not the full set of genes [23] (i.e., prompt contexts) to estimate the probability that different species (LLMs) share the same alleles (tokens) as completion of these specific contexts. Then the formula in Equation 1 is employed to get the final distance matrix. For a given gene $g$ and allele $a$, the proportion of the population with allele $a$ for gene $g$ corresponds to $LLM(a|g)$.

The algorithm is illustrated in Figure 1. The initial step involves collecting model outputs to contexts (genes). Given a set of LLMs, a set of ’genes’, and the specified number of individuals in each population (i.e., the number of times the model is queried on each gene refered to as the number of probes) as $N$, the models are queried for a single token $N$ times. This process generates the matrix $P$, which serves as an approximation of $P(a|g)$, the proportion of the population with allele $a$ to gene $g$. Subsequently, based on this approximation, the similarity matrix $S$ is computed using the Nei genetic distance formula [41] depicted in Equation 1. The pseudo code of PhyloLM can be found in Algorithm 1 in Appendix C.

The algorithm’s essence lies in allowing Large Language Models (LLMs) to complete contexts (’genes’) and assessing the differences in their completions to estimate model distances. The resulting distance matrix reflects the average differences between models on selected genes. For effective differentiation, we require a set of ’genes’ for which completions are not too different between models (otherwise no allele will match and the distance will be maximal between all models) but not too similar (otherwise all models will appear to be the same). The ideal "sweet spot" is somewhere halfway and requires selecting ’genes’ with moderate completion variance between models.

Furthermore, contexts (’genes’) which are very likely to belong to the training data can suffer from contamination issues and generate very low variance²²2To understand this point, imagine using "May the force be with" as context. All models will complete this sentence with "you", thus making impossible establishing distance matrices between them. To obviate this issue, we used contexts (or a ’gene’ set) taken from recent test benchmarks because, in principle, LLMs shouldn’t be trained on this data. To further assess the robustness of our approach and study the impact of the choice of the set of ’genes’, we took our contexts from two different test sets: open-web-math [38] and MBXP [3]. They address different capabilities of LLMs: reasoning and coding, respectively, which are very relevant in recent LLM-related research and are therefore likely to deliver useful results.

The exact selection of contexts from the benchmarks consisted of randomly and uniformly selecting lines from the solution column in the datasets and truncating the text randomly between the 20th and 100th characters to leave it open for LLMs to complete the sentence given a small, but significant, context. In essence, we believe that truncating the contexts to a medium size strikes a balance: it should ensures that the context is sufficiently long to guide the generation process, encouraging Large Language Models (LLMs) to produce similar tokens, while also preventing the emergence of patterns that might lead all LLMs to all generate the same token. ’Gene’ examples are shown in Appendix A.

We devised two complementary analyses to estimate the right hyperparameters to run PhyloLM. The hyperparameters are the ’gene’ set, the number of probes and sampling parameters from the LLM (see Appendix B). Testing the gene set is more difficult as testing thousands of different combinations of genes would come at a very expensive cost. Thus we limited ourselves at 2 parameters of the gene set : the topic (math and code in this paper) and the size of the gene set $G$. In this section we will investigate the impact of $G$ and $N$ in the math gene set, the results for the code gene set are in Appendix D.

First we investigate how $G$ and $N$ affect the variability of the distance matrix, namely how much the similarity matrix changes between different estimations. We focus on similarity matrices (the matrix $S$ in Equation 1) instead of distance matrices at this point as they are bounded in [0,1] making them a lot easier to plot and compare. Then, once the variance is controlled, what combination of $G$ and $N$ approximate reasonably well a very high $G^{\prime}$ and $N^{\prime}$ distance matrix.

To assess the impact of the number of contexts (’genes’) $G$ and the number of probes/individuals $N$ for each dataset, the algorithm was executed across a range of gene set sizes $G$ (varying between 16 and 256 genes per run) and individuals $N$ (ranging from 8 to 128) building similarity matrices. This optimization process, aimed at testing the best values for the algorithm hyperparameters, is particularly computationally expensive. Therefore it was only run on the 5 smallest OPENAI models (ada,babbage,text-ada-001,text-babbage-001 and babbage-002), in order to minimize the costs. Thus similarity matrices in this section are $5\times 5$ making it an estimate of what could be a larger distance matrix at a very low cost.

To investigate the variability of PhyloLM for different combination of hyperparameters, we composed 8 sets of genes of size $G$, each with different genes. Each set of gene is probed $N$ times to build a similarity matrix $S_{G,N,i}$, $i\in[|0,8\|]$ representing the independant set of genes of size $G$ used to generate the matrix with $N$ probes. A variance computation over this set of matrices is finally performed yielding a matrix V containing the variance of each distance between 2 models : ${V_{G,N}}^{2}=\frac{1}{8}\sum_{i}\left(S_{G,N,i}-\left(\frac{1}{8}\sum_{j}{S_{G,N,j}}\right)\right)^{2}$. The square operator is applied coefficient by coefficient. The final variability score is the mean value of the coefficients in the matrix $v_{G,N}=\mu\left(\sqrt{{V_{G,N}}^{2}}\right)$.

Then we investigated the impact of these hyperparameters when trying to approximate a high precision matrix. For this purpose, we compute the variance around a very expensive distance matrix $S_{G^{\prime},N^{\prime}}$ with $G^{\prime}=2048$ and $N^{\prime}=128$. The gene set for the high precision matrix is independent from the lower size set of genes used to estimate it. The formula to compute this variance around the high precision matrix is ${{V^{\prime}}_{G,N}}^{2}=\frac{1}{8}\sum_{i}\left(S_{G,N,i}-S_{G^{\prime},N^{\prime}}\right)^{2}$. The final metric is the mean value in the matrix ${v^{\prime}}_{G,N}=\mu\left(\sqrt{{{V^{\prime}}_{G,N}}^{2}}\right)$.

In situations where models do not share the same tokenizer, comparing only the first alleles generated can pose challenges. For instance, if the context is "The president of the US is Joe," and one model could complete with "Biden" in one token while another could complete with "Bi" "den" in two tokens they would be considered as different alleles while both LLM meant the same completion.

To mitigate this issue of tokenizer alignment, a proxy approach was employed by only using the first 4 characters of the generated text instead of the first token. Practically, each model was instructed to generate at least 4 tokens (tokens are at least 1 character long) and the comparison focused on the first 4 characters in the concatenation of these tokens. In the previous example, the word "Biden" generated in one token or in two ("Bi" and "den") would have been considered as the same response, because the first 4 characters ("Bide") constitute the same ’allele’, despite having being tokenized differently. An example of the results of such a proxy approach is presented in Appendix A.

From a distance matrix obtained by the phylogenetic algorithm it is usual to plot dendrograms representing a possible evolution between the entities in the distance matrix. For this purpose many different algorithms exist and we chose the Neighbour Joining (NJ) technique [39] for its simplicity, efficiency and being a common choice in genetics. We plotted unrooted trees as they are easier to make figures that fit in a paper and are more adapted to LLM evolution than rooted ones. The analysis of the resulting dendrograms also allowed us to validate the capability of our algorithm to predict actual relationship between LLMs in cases where the ground truth is known.

We explored whether genetic distance can predict model performance by using logistic regression to estimate benchmark scores of large language models based on their similarity to other models. Due to the high dimensionality of the similarity matrix, we reduced the input dimensions to 15 using Independent Component Analysis (ICA), resulting in 15 parameters to learn from approximately 100 data points per fit. We then applied a sigmoid function to the output to scale the predictions between 0 and 1, corresponding to benchmark scores ranging from 0% to 100%. Since benchmark scores can be highly correlated within a family of models, we employed a leave-one-family-out method. This involved training the regressor on all families but one and testing it on the excluded family. A Mean Squared Error loss was used with an Adam optimizer (learning rate of $10^{-3}$).

We tested the benchmarks available on the hugging face open llm leaderboard which includes MMLU, ARC, Hellaswag, TruthfulQA, Winogrande and GSM8k [25] and only included open access models for which the scores are available on the leaderboard. Thus we didn’t include proprietary models in this study as, as explained in later sections, distance computation is slightly biased for these models and benchmark scores are not obtained in the same conditions as in the leaderboard (number of shots, CoT, …). The benchmarks used for the ’gene’ set were distinct from these benchmarks to avoid any type of contamination between the ’alleles’ used to generate genetic distances and the performance of the models in the considered benchmark tasks.

We first ran the hyperparameters’ optimization process explained in Methods2.3 and plotted the results in figure 2(a) left side. This graph shows a clear decrease in the variability as the number of ’genes’, $G$ grows with almost no effect from $N$. This is interesting : it seems that having different sets of ’genes’ doesn’t appear to change the similarity matrix as long as there are enough of them (at least in the open-web-math and mbxp dataset - see Appendix D for the results on the code set of ’genes’).

However this method doesn’t make it possible to find a good $N$, indeed, the probability for two models to generate the same token in the same context in only one try is quite low. Therefore, a very low $N$ will make all models appear particularly different making the similarity matrix look like the identity matrix yielding unsatisfactory results despite having a low variance. Thus having a $N$ high enough is required to get a useful similarity matrix and we need to find a better metric but how to choose it ?

We have just seen that $G$ monitors the variability of the matrix (variability parameter), thus a similarity matrix with a very high $G$ should be particularly stable across different sets of genes. Consequently, we considered a very highly over-parametrised similarity matrix (refered to as high precision matrix), with a large set of genes $G^{\prime}=2048$ and very high number of probes $N^{\prime}=128$ (to be a good approximation of the function $P$). We then compared more modestly parametrized similarity matrices to study how hyperparameters $G$ and $N$ influence the difference between these lower precision matrices to the high precision matrix on average (see Methods 2.3 for the computational details). This new metric should penalize having a low $N$ leading to similarity matrices close to the identity matrix and may yield more satisfying results.

As shown in Figure 2(b), while increasing the number of genes still seems to approximate better high precision matrix, this time, the number of probes is also very important. Indeed, for each value of $N$, the performance saturates from some $G$ value making less and less improvement when $G$ increases. Thus, this figure gives an optimal $G$ for a given $N$ in order to approximate the high precision matrix efficiently with a low cost. The total cost of the algorithm in tokens being proportional to $G\times N$, we found a good tradeoff between variance and precision around $G=128$ and $N=32$.

The estimated cost to run the algorithm per model is therefore $128\,\mbox{genes}\times 32\,\mbox{probes}=4096$ queries of $\approx 20$ tokens. As a point of reference, conducting the MMLU benchmark requires around 14,000 queries on significantly longer prompts ($\approx 70$ tokens each), making PhyloLM approximately 10 times less expensive in terms of the number of tokens required.

We first examine the results of PhyloLM by analyzing the resulting phylogenic trees (materialized as dendrograms). However, before dwelling into the results, an important point to understand is that, in genetics, branches in the tree show probable speciation events that occured in the past, when from an extinct common ancestor, two (or more) current species (leaves of the tree) emerged. When looking at LLMs, ’common ancestors’ are not extinct, but rather among the studied ’populations’. Take for instance Mistral 7B that is the common ancestor of OpenChat3.5 and Zephyr 7B Alpha, but still included in our analysis. Oblivious of this difference, the dendrogram plotting method will put all models at the ’leaves’ of the tree, while, in fact, some of them (such as Mistral 7B) should be at a speciation node. As such, without additional information about which model is at a node, it is difficult to interpret them in the same way as in genetics. Without this important phylogenetic assumption, one has to bear in mind that what matters (and should be compared with the ground truth) is their relative distance and position when evaluating the dendrograms resulting from the phylogenetic analysis of LLMs. Indeed the distance between two models is represented by the distance from their respective leaves in the dendrogram.

To investigate the capabilities of PhyloLM, let’s first start by respecting this assumption by looking at a set of 9 models from the Mistral family whose relationships are known because transparently disclosed by their creators. Out of these 9 models, 5 are leaves in the ground truth dendrogram [2, 34, 55, 54, 50]. Running PhyloLM on these 5 models getting the distance matrix between them and finally plotting the NJTree we perfectly get back the ground truth phylogenetic tree (see Fig 3) validating the method. These rooted trees are not necessarily very stable as the NJ algorithm makes an unrooted tree of the evolution but then has to choose the root. In Appendix D we show that, on the code genome, the root has been mistakenly attributed to model 3 while the structure of the tree is right. That is why we prefer to plot unrooted trees in the rest of this paper.

We then investigated whether the genetic distance metric can be used to predict the abilities of language models. As such we used the benchmark scores from the Huggingface open LLM leaderboard. The results indicate that the prediction correlates with the true score of the models (Figure 5 (a)). Indeed, we found that the Pearson’s correlation coefficients (r) of the correlation between the true scores and the predicted ones was positive and significant for all benchmarks and regardless of the set of “genes” used to make the prediction (mean±sem: 0.68±0.04; Student’s t-test again zero: t(11)=16.0, p<0.001; Figure 5 (b)). In other terms, within benchmarks and across families, the phylogenetic distance metric allowed us to predict on average 48.2±0.03% of the variance of the between-model benchmark performance. In a control analysis, we also verified that significant correlation was also achieved within families, thus eliminating the possibility that significant prediction in the previous analysis was driven by our metric simply capturing the fact that different families have different levels of performance on average. To do so, we calculated the Pearson correlation between the true and the predicted scores per benchmark and within each family separately. The results indicate that, even though for some combinations of families and benchmarks, we obtained small or negative correlation coefficients (which is unsurprising, since these correlations were sometimes calculated across very few data points), also in this case, the results were in average positive and significant difference from zero (0.64±0.05; t(107)= 20.7, p<0.001; Figure 5 (c)). Within families, the variance explained by our method amounted to 52.2±0.03% on average, thus indicating that our metric achieved good predictive power even when drastically increasing the level of granularity.