SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation

The theoretical framework of DA is well established [2, 41, 42]. The main results highlight that the performance discrepancy of an estimator between the source and target domains is linked to the divergence between both distributions. This has motivated the majority of DA methods to search for a universal (or domain invariant) predictor by minimizing the divergence between the two domains through the adaptation of the distributions. This is done in practice by modeling and estimating the shift between the source and target distributions and then compensating for this shift before training a predictor.

A wide variety of shifts between the source and target distributions are possible. They are usually expressed as a relation between the joint distributions $P^{s}(x,y)=P^{s}(x|y)P^{s}_{\mathcal{Y}}(y)=P^{s}(y|x)P^{s}_{\mathcal{X}}(x)$ in the source domain and $P^{t}(x,y)=P^{t}(x|y)P^{t}_{\mathcal{Y}}(y)=P^{t}(y|x)P^{t}_{\mathcal{X}}(x)$ in the target domain. We now discuss the main types of shifts and the strategies proposed in the literature to mitigate them. Figure 1 illustrates these shifts.

In Covariate shift the conditionals probabilities are equal (i.e., $P^{s}(y|x)=P^{t}(y|x)$), but the feature marginals change (i.e., $P^{s}_{\mathcal{X}}(x)\neq P^{t}_{\mathcal{X}}(x)$). Target shift is similar, but the label marginals change $P^{s}_{\mathcal{Y}}(y)\neq P^{t}_{\mathcal{Y}}(y)$ while the conditionals are preserved. For classification problems, it corresponds to a change in the proportion of the classes between the two domains. Both of those shifts can be compensated by reweighting methods that assign different weights to the samples of the source domain to make it closer to the target domain [50, 46].

In Conditional shift, conditional probabilities differ between domain (i.e., $P^{s}(x|y)\neq P^{t}(x|y)$ or $P^{s}(y|x)\neq P^{t}(y|x)$). This shift is typically harder to compensate for, necessitating explicit modeling to address it effectively. For instance, several approaches model the shift as a mapping $m$ between the source and target domain such that $P^{s}(y|m(x))=P^{t}(y|x)$ [52, 7]. The estimated mapping is then applied to the source data before training a predictor.

Subspace shift, also known as domain invariant representation, assumes that while probabilities are different between the domains, there exists a subspace $\mathcal{Z}$ and a function $\phi:\mathcal{X}\to\mathcal{Z}$ such that $P^{s}(y|\phi(x))=P^{t}(y|\phi(x))$. This implies that a classifier trained on this subspace will perform well across both domains. Subspace methods aim to identify the subspace $\mathcal{Z}$ and the function $\phi$, as developed in [39, 12]. Note that, as discussed in the introduction, a natural extension of this idea is to learn a non-linear invariant subspace using deep learning [15, 53].

In the literature, few papers propose realistic DA scorers to validate the parameters of the methods, i.e., unsupervised scorers that do not require target labels. The Importance Weighted (IW) scorer [49] computes the score as a reweighted accuracy on labeled sources data. The Deep Embedded Validation (DEV) [59] can be seen as an IW in the latent space with a variance reduction strategy. DEV was originally proposed for deep learning models but can be used on shallow DA methods that compute features from the data (mapping/subspaces). The Prediction Entropy (PE) scorer [37] measures the uncertainty associated with model predictions on the target data. Soft Neighborhood Density (SND) [45] also computes an entropy but on a normalized pairwise similarity matrix between probabilistic predictions on target. The Circular Validation (CircV) scorer [3] performs DA by first adapting the model from the source to the target domain and predicting the target labels. Next, it adapts back from the target to the source using these estimated labels. Performance is measured as the accuracy between the recovered and true source labels.

The model selection problem in DA has been widely discussed in the literature. Yet, this literature constitutes a subfield of DA and has seldom been used to validate new DA methods. Indeed, there is no consensus on the best validation strategy and many papers do not properly validate their methods, leading to over-estimated performances. Some authors do not discuss the validation procedure [50, 46] or consider fixed hyperparameters [21]. While some methods rely on custom validation technique [51], others use cross-validation, either on the source or the target [52, 7], or alternatively other validation strategies proposed in the literature [6, 3]. A complete picture of the model selection procedures used to validate the methods considered in SKADA-Bench in their original papers is presented in Table A.1 in Appendix. The goal of SKADA-Bench is therefore to constitute a dedicated benchmark to compare scorers from the literature and report performances that can be expected in real use cases for the considered methods.

We propose a nested loop cross-validation procedure, depicted in Figure 2. First, the source and target data are split into multiple outer test and train sets (outer loop in Figure 2). The test sets are kept to compute the final accuracy for both the source and target domains. For each split in the outer loop, we use a nested loop to select the DA methods’ parameters. Here, the training sets are further divided into nested train and validation sets (nested loop in Figure 2). Note that no labels are available for the target nested train and validation sets in this loop. The target training set is used to train the DA method, while the target validation set allows to compute the unsupervised score and select the best model.
For both loops, the data is split randomly 5 times using stratified sampling with an 80%/20% train/test split. For one given method, we evaluate all the unsupervised scorers discussed earlier, as well as a supervised scorer that uses target labels, over all the nested splits. After averaging, the scores over the splits, the best hyperparameters are selected according to each scorer and then used to train a final classifier on the outer training sets. Although the supervised scorer cannot be used in practice, it is included in our results to actually evaluate the performance drop due to the absence of target labels. To limit complexity and perform a fair comparison of the methods, we set a timeout of 4 hours for performing the nested loop.

Existing domain adaptation methods typically rely on either a base estimator trained on the adapted data or an iterative estimation process to adapt this estimator to the target data. The choice of the base estimator is crucial, as it significantly impacts the final performance. Before validating the hyperparameters of the DA methods, we determined the best estimator for each dataset using a grid-search on the source data. We tested multiple hyperparameters for Logistic Regression, SVM with RBF kernel, and XGBoost [5], selecting the ones that maximize the average accuracy on the source test sets. Note that for some methods that specifically require an SVM estimator (i.e., JDOT and DASVM), we only validate SVM as the base estimator. We validated the base estimator separately from the DA methods parameters to reduce computational complexity and avoid too complex hyperparameter grids that can compromise the reliability of DA scorers.

For all methods, we select the best validation scorer as the one that maximizes the averaged accuracy on the target domains for all real datasets. This provides a reasonable and actionable choice of scorer for each DA method for practitioners. For all methods and datasets, we perform a paired Wilcoxon signed-rank test at the $0.05$ level to detect significant gain or drop in performance with respect to the no DA approach, denoted by “Train Src” in the following. The test is done using the accuracy measures of the DA method with the selected scorer and the Train Src for all shifts and outer splits, ensuring between 10 and 60 values depending on the dataset.

The benchmark code will be made available on GitHub upon publication of the paper.^***Our code is available in supplementary materials. Our benchmark is implemented following the benchopt framework [35], which provides standardized ways of organizing and running benchmarks for ML in Python. This framework facilitates reproducing the benchmark’s results, with tools to install the dependencies, run the methods in parallel, or cache the results to prevent redundant computations. It also makes it easy to extend the benchmark with additional datasets and methods, enabling it to evolve to account for the advances in the field. In the supplementary materials, we provide examples demonstrating how to add DA methods or datasets to the benchmark. Using this framework, we aim to make SKADA-Bench a reference benchmark to evaluate new DA methods in realistic scenarios with valid performance estimations.

These methods aim to reweight the source data to make it closer to the target data. The weights are estimated using different methods such as kernel density estimation (Dens. RW) [50], Gaussian estimation (Gauss. RW) [46], discriminative estimation (Discr. RW) [46], or nearest-neighbors (NN RW) [30]. Other reweighting estimate weights by minimizing a divergence between the source and target distributions such as Kullback-Leibler Importance Estimation Procedure (KLIEP) [51] or Kernel Mean Matching (KMM) [21]. Finally, we also include the MMDTarS method [60] that uses a Maximum Mean Discrepancy (MMD) to estimate the weights under the target shift hypothesis.

Mapping methods. These methods aim to find a mapping between the source and target data that minimizes the distribution shift. The Correlation Alignment method (CORAL) [52] aligns the second-order statistics of source and target distributions. The Maximum Mean Discrepancy (MMD-LS) method [60] minimizes the MMD to estimate an affine Location-Scale mapping. Finally, the Optimal Transport (OT) mapping methods [7] use the optimal transport plan to align with a non-linear mapping of the source and target distributions with exact OT (MapOT), entropic regularization (EntOT), or class-based regularization (ClassRegOT). Finally, the Linear OT method [13] uses a linear mapping to align the source and target distributions, assuming Gaussian distributions.

Subspace methods. These methods aim to learn a subspace where the source and target data have the same distribution. The Transfer Component Analysis (TCA) method [39] searches for a kernel embedding that minimizes the MMD divergence between the domains while preserving the variance. The Subspace Alignment (SA) method [12] aims to learn a subspace where the source and target have their covariance matrices aligned. The Transfer Subspace Learning (TSL) method [47] aims to learn a subspace using classical supervised loss functions on the source (e.g., PCA, Fisher LDA) but regularized so that the source and target data have the same distribution once projected on the subspace. Finally, the Joint Principal Component Analysis (JPCA) method is a simple baseline that concatenates source and target data before applying a PCA on all data.

Others. We also include other methods that do not fit into the previous categories. The Domain Adaptation SVM (DASVM) method [3] is a self-labeling method that iteratively updates SVM estimators by adding new target samples with predicted labels and removing source samples. The Joint Distribution Optimal Transport (JDOT) method [6] aims to learn a target predictor that minimizes an OT loss between the joint source and target distributions. The Optimal Transport Label Propagation (OTLabelProp) method [48] uses the optimal transport plan to propagate labels from the source to the target domain.

The objective of the simulated datasets is to evaluate the performance of the DA methods under different types of shifts. Knowing that multiple DA methods have been built to handle specific shifts, evaluating them with this dataset will demonstrate whether they perform as expected and if they are properly validated.
The 4 simulated shifts in 2D, covariate (Cov. shift), target (Tar. shift) conditional (Cond. shift) and Subspace (Sub. shift) shift are illustrated in Figure 1. The source domain is represented by two non-linearly separable classes generated from one large and several smaller Gaussian blobs. In the experiments, the level of noise has been adjusted from Figure 1 to make the problem more difficult. For the subspace shift scenario, the source domain consists of one class represented by a large Gaussian blob and another class comprising Gaussian blobs positioned along the sides of the large one. The target domain is flipped along the diagonal, making the task challenging in the original space but feasible upon diagonal projection.

The real-world datasets used in our benchmark are summarized in Table 1. We select $8$ datasets from different modalities and tasks: Computer Vision (CV) with Office31 [26], Office Home [55], and MNIST/USPS [28], Natural Language Processing (NLP) with 20Newsgroup [27] and Amazon Review [33], Tabular Data with Mushrooms [8] and Phishing [34], and Biosignals with BCI Competition IV [54]. The datasets are chosen to represent a wide range of shifts and to evaluate the performance of the methods on different types of data.
The datasets are preprocessed with feature extraction to ensure reasonable performance when trained on each domain. For example, images are embedded using pre-trained models followed by a PCA (except MNIST/USPS where only PCA is used), and textual data is embedded using Large Language Models (LLM) [43, 57] before applying a PCA. The tabular data are one-hot encoded to transform categorical data into numerical data. The biosignals from Brain-Computer Interface (BCI) data are embedded using the state-of-the-art tangent space representation proposed in [1]. The datasets are split into pairs of source and target domains totaling $51$ adaptation tasks in the benchmark. More details about the datasets and pre-processing are available in Appendix B.

First, we report the realistic performances of the different methods when using their selected scorer on the different datasets in Table 2. The cells showcasing a significant change in performance with the Wilcoxon test are highlighted with colors. Blue indicates an increase in performance, while red indicates a loss. The intensity of the color corresponds to the magnitude of the gain or loss - the darker the shade, the larger the positive or negative change. Cells with a NA values indicate that the method was not applicable to the dataset (DASVM is limited to binary classification) or that the method has reached a timeout. We also report the best scorer and the average rank of the methods for all datasets. In addition to Table 2 providing realistic performance estimations, we also report in Table D.12 (Appendix D) the results when using the non-realistic supervised scorer.

On simulated data with known shifts (i.e., underlined datasets), DA methods tend to show a small but significant gain on the shift they were designed for. However, they rarely reach the Train Tgt performances, and as expected, the methods perform poorly on the shift they were not designed for. For Cov. shift, the improvement with reweighting methods is very limited. We believe that using a complex base estimator (SVM with an RBF kernel) enables us to train an estimator that works well on both source and target, leading to less impressive results. The Appendix D contains detailed tables for two other base estimators (i.e., Logistic Regression in Table D.9 and XGBoost in Table D.11). These tables show that the gain is larger for Logistic Regression, a linear classifier, although the overall performance is lower. Note that in the literature, the base estimator used is often a simple method, which can explain the performance gains reported in various papers.

On the real-life datasets, we observe that DA methods do not consistently outperform the Train Src approach. The only exceptions are the LinOT and CORAL methods, which either result in a significant improvement or perform comparably to the Train Src approach. OT-based mapping methods show gains on MNIST/USPS and 20NewsGroups. Other methods, such as ClassRegOT, MMD-LS, JPCA, and SA, occasionally show better accuracy than Train Src. However, interestingly, DA methods fail to improve performance on datasets like Office31, AmazonReview, and Phishing. For the AmazonReview dataset, Train Src performs exceptionally well, possibly because the selected LLM has been partially trained on similar Amazon review data, leading to invariant representations [57].
Looking at the average rank of the methods, we observe that the Train Src approach has a relatively small rank of 9.75 out of 20, while the best-rank DA methods are LinOT with a rank of 5.33 and CORAL with 7.10. Other methods with a lower rank than Train Src are ClassRegOT, MMDTarS, JPCA, and SA.

We observe that the best scorer differs across methods, Circular Validation has been selected 12 out of 20 times as the best scorer, followed by Importance Weighting 5 out of 20 times. Table D.12 in the supplementary material provides the accuracy results with the supervised scorer. It is worth noting that the supervised scorer generally outperforms the unsupervised ones, and several methods significantly outperform Train Src in each dataset. It is crucial to choose the model realistically to avoid producing overly optimistic results, as many data analysis papers have done (see able A.1).

These results show the methods’ sensitivity to parameter selections and the difficulty of using realistic scorers. This might also explain why DA methods are not widely used in practice: they are very difficult to tune and might decrease performances compared with no adaptation.