Mind the Gap: Measuring Generalization Performance Across Multiple Objectives

Abstract

Modern machine learning models are often constructed taking into account multiple objectives, e.g., minimizing inference time while also maximizing accuracy. Multi-objective hyperparameter optimization (MHPO) algorithms return such candidate models, and the approximation of the Pareto front is used to assess their performance. In practice, we also want to measure generalization when moving from the validation to the test set. However, some of the models might no longer be Pareto-optimal which makes it unclear how to quantify the performance of the MHPO method when evaluated on the test set. To resolve this, we provide a novel evaluation protocol that allows measuring the generalization performance of MHPO methods and studying its capabilities for comparing two optimization experiments.

A Experimental Details

Random Forest		Linear Model
Hyperparameter name	Search space	Hyperparameter name	Search Space
criterion	[gini, entropy]	penalty	[l2, l1, elasticnet]
bootstrap	[True, False]	alpha	$[1e-6, 1e-2]$, log
max_features	[0.0, 1.0]	l1 ratio	[0.0, 1.0]
min_samples_split	[2, 20]	fit_intercept	[True, False]
min_samples_leaf	[1, 20]	eta0	$[1e-7, 1e-1]$
pos_class_weight exponent	$[-7, 7]$	pos_class_weight exp	$[-7, 7]$

We provide the random forest and linear model search spaces in Table A. We fit the linear model with stochastic gradient descent and use an adaptive learning rate and minimize the log loss (please see the scikit-learn (Pedregosa et al., 2011) documentation for a description of these). Because we are dealing with unbalanced data, we consider the class weights as a hyperparameter and tune the weight of the minority (positive) class in the range of $[2^{-7}, 2^7]$ on a log-scale (Horn and Bischl, 2011; Konen et al., 2016). To deal with categorical features, we use one hot encoding. We transform the features for the linear models using a quantile transformer with a normal output distribution.

We use the German credit dataset (Dua and Graff, 2017) because it is relatively small, leading to high variance in the algorithm performance, and unbalanced. We downloaded the dataset from OpenML (Vanschoren et al., 2014) using the OpenML-Python API (Feurer et al., 2021) as task ID 31, but conducted our own 60/20/20 split. It is a binary classification problem with 30% positive samples. The dataset has 1000 samples and 20 features. Out of the 20 features, 13 are categorical. The dataset contains no missing values.