An Experience in the Evaluation of Fault Prediction

Abstract

Background. ROC (Receiver Operating Characteristic) curves are widely used to represent the performance (i.e., degree of correctness) of fault proneness models. AUC, the Area Under the ROC Curve is a quite popular performance metric, which summarizes into a single number the goodness of the predictions represented by the ROC curve. Alternative techniques have been proposed for evaluating the performance represented by a ROC curve: among these are RRA (Ratio of Relevant Areas) and $\phi$ (alias Matthews Correlation Coefficient).

Objectives. In this paper, we aim at evaluating AUC as a performance metric, also with respect to alternative proposals.

Method. We carry out an empirical study by replicating a previously published fault prediction study and measuring the performance of the obtained faultiness models using AUC, RRA, and a recently proposed way of relating a specific kind of ROC curves to $\phi$, based on iso-$\phi$ ROC curves, i.e., ROC curves with constant $\phi$. We take into account prevalence, i.e., the proportion of faulty modules in the dataset that is the object of predictions.

Results. AUC appears to provide indications that are concordant with $\phi$ for fairly balanced datasets, while it is much more optimistic than $\phi$ for quite imbalanced datasets. RRA’s indications appear to be moderately affected by the degree of balance in a dataset. In addition, RRA appears to agree with $\phi$.

Conclusions. Based on the collected evidence, AUC does not seem to be suitable for evaluating the performance of fault proneness models when used with imbalanced datasets. In these cases, using RRA can be a better choice. At any rate, more research is needed to generalize these conclusions.

Partly supported by Fondo di Ricerca d’Ateneo dell’Università degli Studi dell’Insubria.