Detecting Risk of Biased Output with Balance Measures

The large availability of data, in conjunction with the widespread use of predictive, classification, and ranking models, has fueled the ongoing mass digitization of organizational processes in our societies [3]. This is especially true for decision-making processes, which are rapidly turning into automated data-driven decision-making systems in a variety of sectors, both in private and public organizations. Such processes range from predicting debt repayment capability to identifying the best candidates for a job position, from detecting social welfare frauds to suggesting which university to attend; just to mention a few cases [4]. Advantages for using these systems concern scalability of the operations and consequent economic efficiency, as well as the removal of human subjectivity and errors. Though the benefits materialize only if the underlying data are of high quality, otherwise errors could lead to relevant extra costs [18] and also give rise to serious ethical issues: several studies showed that automated data-driven processes replicate or even amplify the same bias of our society, producing systematic discrimination against the weakest people, and exacerbating existing inequalities [16]. A recurring cause for unintended but nevertheless dramatic consequence is the use of biased data. From a data engineering perspective, this means imbalanced data, i.e., the condition of uneven distribution of data among the classes of a given attribute, which causes highly heterogeneous accuracy across the classifications [11]. Imbalance can origin from errors or limitations in the data collection, design, and operations, or simply from the reality that the data itself reproduce. When the objects of automated decisions are individuals, such disparate performance of the algorithm represents in practice a systematic discriminatory behavior that causes relevant social, legal, and ethical issues [2].

In this article, we investigate whether, and to which extent, it is possible to assess the risk of unfairness in software output by measuring the imbalance of protected attributes in training data. We describe the design of the proof of concept in Section 2 and results in Section 3. Then, we position our work in the literature in Section 4, and we highlight the limitations of the study in Section 5; we conclude with a few final remarks in Section 6.

On the basis of the motivations presented above we formulated the following research question:

The research question relies on the following definitions:

With the goal of exploring the research question, we set up a method to deliver a proof of concept:

To explore our RQ and check whether lower levels of balance—as detected by the selected measures—correspond to higher unfairness levels, we finally assessed the relationship between the unfairness measures and the balance measures.

Before addressing the main research question, we performed a sanity check by observing the behavior of the balance measures as the mutation parameter p varies. Figure 1 reports the average values for different balance measures and datasets. We observe an increasing trend of all the balance measures w.r.t. increasing p, in all training sets and test sets. More in detail, Gini and Shannon indexes have a super-linear increase; Simpson index is closer to a linear trend; finally, IR index has a sub-linear increase until \(2/3\) of the course and then it turns to have a slight super-linear increase. This observation confirms the ability of the mutation approach to generate synthetic datasets that spread the whole range of conventional balance measures.

Figure 2 reports the variation of the five fairness criteria (Y-axis) w.r.t. the increase of balance measures (X-axis). The lines are smoothed regression of the individual mutations. For sake of legibility, we omitted Gini since it is very similar to Shannon. We can observe from the curves that very low levels of balance—roughly in the range \([0, 0.15]\) and up to 0.50 in a few cases—correspond to higher levels of unfairness. As shown in the preliminary results, the indexes react slightly differently to different levels of balance: as a consequence, the distinct unfairness criteria reflect different levels of balance in a slightly different way. By looking at the single fairness criteria, as well as at the specific trend lines in Figure 2, we observe that:

On the basis of these observations and within the limits of this proof concept, we positively answer our initial research question. Moreover, we can identify tentative thresholds of balance measures and the following practical recommendation:

Our contribution can be located in the main corpus of researches on algorithmic bias and fairness. While most of the literature focus on the outputs of ADM systems, we focus on the inputs and processes, following a direction suggested by several recent studies (e.g., [5, 17] and [8]). Our approach has its theoretical and methodological foundations in the ISO/IEC standards on data quality measurement [9] and on risk management [10]: for space reasons, we can not analytically report on all the relations between our proposed approach and the two ISO/IEC standards, which can be found in [19]. This study expands the research reported in [20]: herein, we introduced a mutation technique to generate a number of derived synthetic datasets having different levels of balance, instead of relying on a few exemplar distributions as done in the previous study. We applied a similar technique also in [14], but not specifically to binary attributes as done here. A further novelty in this article is the computation of the Sufficiency criterion of fairness, in addition to Independence and Separation.

An approach similar to ours and with a wider scope is the work of Matsumoto and Ema [13], who proposed a risk chain model for risk reduction in Artificial Intelligence (AI) services, named RCM. The authors consider data imbalance as a risk factor; however, they do not indicate measures for it: as a consequence, our quantitative approach to measure (im)balance can reasonably fit into their framework. Our work is also complementary to the existing toolkits for bias detection and mitigation [12], since therein the proposed measures of balance are not taken into consideration yet.

The limited number of datasets that has been taken into account, as well as the set of balance measures constitute notable limitations to our study. More datasets and more metrics are necessary to generalize the findings of this exploratory work, also by including measures for non-categorical data. In addition, as the choice of the balance measure has a relevant impact on the threshold to consider as risky, in-depth sensitivity analyses on the thresholds should improve the reliability of the findings presented here.

Furthermore, as we ran the binomial logistic regression, all the limitations of this classification model hold, most notably the two assumptions of limited or no multi-collinearity between independent variables, and of linearity between the dependent variable and the independent variables. Applying more classification algorithms (each with different parameters) would improve the external validity of the relationship we found between balance and unfairness in the classification output, and would help to identify how the different types of classification algorithms propagate the imbalance from the training set to the output. Concerning the mutations of the datasets used, the set of values of parameter p could be enriched with further entries, to track the relationship with unfairness in a more granular way. In addition, other kinds of mutation techniques could be also considered by adopting different pre-processing methods.

In this article, we evaluated whether imbalanced distributions of a binary protected attribute in the training data can lead to discriminatory output of ADM systems. We selected four balance measures (the Gini, Simpson, Shannon, and Imbalance Ratio indexes, normalized to share the same range of values and semantics), we applied them to training sets, and tested their ability to detect unfairness occurring in classification tasks. Overall, the results showed that our approach is suitable for the proposed goal; however, the choice of the balance measure has a relevant impact on the threshold to consider as risky. Hence, further work shall be devoted to thorough and systematic investigation of the thresholds to be used, also in combination with different prediction models and mutation techniques.