Dynamic Fairness-aware Recommendation Through Multi-agent Social Choice

Our research in fairness examines concepts inspired by the application context of Kiva Microloans, a non-profit organization that offers a platform (Kiva.org) for crowd-sourcing the funding of microloans, mostly in the developing world. Kiva’s users (lenders) choose among the loan opportunities offered on the platform; microloans from multiple lenders that are aggregated and distributed through third party non-governmental organizations around the world. Figure 1 shows an example of a listing for a loan that a user might choose to support.

Kiva Microloans's mission specifically includes considerations of “global financial inclusion”; as such, incorporating fairness in its recommendation of loans to potential users/lenders is a key goal. To a large part, these fairness issues are around ensuring that all of the borrower’s whose loans are shown on Kiva have a fair chance to receive funding and counteracting biases that lenders might have about who is “worthy” of support. There are many complexities in international development and a number of different ways that fairness might be formulated. This complexity was a key motivator for us to seek an architecture that could integrate multiple heterogeneous concerns into the recommendation process.²

Kiva’s platform, and the motivating setting for our article, is a web- and mobile app-based recommendation system. Users arrive at the Kiva.org website and are served an ordered list of loans that they may choose to fund. Hence, our system outputs a (small) ranked list of items from a larger inventory. We will use Kiva’s platform as an example throughout this article. However, our findings and associated implementations are not specific to this setting.

Fairness in recommender systems is a broad topic that has received significant attention in recent years due to number, variety, and impact of these systems. As pointed out by Wang et al. [57], recommender systems often allocate scarce social resources including jobs and money, and hence ensuring the fairness of these systems is a critical ethical concern. Two recent surveys of the landscape each review over 150 papers in the area. Wang et al. [57] provide a survey of papers in top conferences that focus on recommendation system fairness. They include a taxonomy of fairness research that includes our views here: multistakeholder and multi-definitional. Indeed, our work here addresses two key directions for future research outlined in the survey, specifically win-win models for fairness and accuracy and fairness in a real system. Deldjoo et al. [18] provide a survey of the fairness in recommendation systems field including many of the definitions and contexts in which the problem are studied. In particular, the authors note that many works are focused on abstract problem representations and that more work with stakeholders is necessary. Our work directly addresses this call.

There have been a number of efforts that explicitly consider the multisided nature of fairness in recommendation and matching platforms. Patro et al. [44] investigate fairness in two-sided matching platforms where there are both producers and consumers. They note, as we do, that optimizing fairness for only one side of the market can lead to very unfair outcomes for the other side of the market. Patro et al. [44] also appeal to the literature on the fair allocation of indivisible goods from the social choice literature [54]. They devise an algorithm that guarantees max-min share fairness of exposure to the producer side of the market and envy-free up to one item to the consumer side of the market [3]. Their work is closest to the allocation phase of SCRUF-D. However, in contrast to our work, they only use exposure on the producer side and relevance on the consumer side as fairness metrics, whereas SCRUF-D is able to capture additional definitions. Also, we note that envy-freeness is only applicable when valuations are shared: a condition not guaranteed in a personalized system. It is possible for a user with unique tastes to receive low utility recommendations and still not prefer another user’s recommendation lists. Also, our fairness formulation extends beyond the users receiving recommendations to providers of recommended items and envy-freeness provides no way to compare users who are recieving different types of benefits from a system. In addition, our fairness definitions are dynamic, a case not considered by Patro et al. [44]. Indeed, in a recent survey paper, Patro et al. [45] provide an overview of some of the current work and challenges in fair recommendation, highlighting the focus of our work, multi-sided, multistakeholder, dynamic, and contextual issues, as key challenges.

Like Patro et al. [44], the work of Sühr et al. [53] investigates fairness in two-sided platforms, specifically those like Uber or Lyft where income opportunities are allocated to drivers. However, unlike our work and the work of Patro et al. [44], Sühr et al. [53] take proportionality as their definition of fairness, specifically proportionality with respect to time in a dynamic setting, and ensure that there is a fair distribution of income to the provider side of the platform. Buet-Golfouse and Utyagulov [9] also tackle the problem of multi-sided fair recommendation, arguing that recommender systems are built on sparse data and that the algorithms must take into account fairness. To this end they propose a regularization term that can be incorporated into the recommendation algorithm itself and account for various biases, e.g., exposure and/or selection bias, depending on the particular definition of the regularization term. The work of Buet-Golfouse and Utyagulov [9] is different from ours in that we make use of post-processing/re-ranking, as it allows for more flexibility with respect to the fairness definitions and allows us to treat the recommendation phase as a black-box, thus our system can be layered with any traditional user/item recommendation algorithm. A post-processing approach also has the advantage that the set of fairness concerns and their relative importance can be adjusted on the fly without model retraining.

Working with large item sets and multiple definitions of fairness can be computationally challenging. For example, Zehlike et al. [62] propose a system that allows for multiple protected groups on the provider side and their system works online, i.e., not in an offline, batch mode. But it is designed more for a general ranking task and is very computationally inefficient for recommendation. Every new recommendation list to be reranked requires the generation of a binary tree of size \(|G|^k\), where G is the number of protected groups and k is the size of the output list.

Freeman et al. [24] investigate what they call dynamic social choice functions in settings where a fixed set of agents select a single item to share over a series of timesteps. The work focuses on overall utility to the agents instead of considering the multiple sides of the recommendation interaction. Their problem is fundamentally a voting problem, since all agents share the result, whereas we are focused on personalized recommendation. Their goal is to optimize the Nash Social Welfare of the set of agents (that remains fixed at each timestep) and present four algorithms to find approximately optimal solutions. This work has a similar flavor to classical online learning/weighting experts problems [16] in the sense that the agent preferences remain fixed and the goal is to learn to satisfy them over a series of timesteps. Similarly, Kaya et al. [30] focus on the problem of group recommendation, which is similar to classical social choice setting of multi-winner voting [67]; i.e., a group of k items are recommended as a shared resource to a set of users, and fairness is defined w.r.t. to this group of users preferences.³ Kaya et al. [30] propose methods that are similar to those found in the multi-winner voting literature that attempt to be fair w.r.t. the individual preferences of the group, and these algorithms are meant to aggregate user preferences, not deliver specific recommendations on a per user basis, a key difference with our work.

Ge et al. [26] investigate the problem of long-term dynamic fairness in recommendation systems. This work, like ours, highlights the need to ensure that fairness is modeled as a temporal concept and not as a static, one off, decision as is often done in batch fairness models [57]. To this end, they propose a framework to ensure fairness of exposure to the producers of items by casting the problem as a constrained Markov Decision Process where the actions are recommendations and the reward function takes into account both utility and exposure. Ge et al. [26] propose a novel actor-critic deep reinforcement learning framework to accomplish this task at the scale of large recommender systems with very large user and item sets. Again, this work fixes definitions of fairness a priori, although their learning methodology may serve as inspiration for an online extension of our allocation stage problems in future work. The subject of the long-term impacts of recommender systems is also considered by Akpinar et al. [1] who investigate the effects of various fairness interventions on the whole system, over time, finding similar effects to those of Ge et al. [26]. This reinforces the decisions in SCRUF-D to take into account the long-term dynamic effects of any fairness intervention.

Reinforcement learning and other online learning methodologies have been proposed for various settings of fair recommendation. Zhang and Wang [64] provide a short position paper proposig using reinforcement learning and an underlying Markov Decision Process to learn priorities and feedback online during user interaction. While this is similar to a typical learning to rank system/framework [63] there are some key differences in their proposal, including learning fairness metrics. While Zhang and Wang do not propose a system, we agree that it is an intriguing direction and methodology to incorporate reinforcement learning (RL) into fair recommendation frameworks. In the learning to rank space, wherein a recommendation system is attempting to learn the (set of) users’ preference function online, as in reinforcement learning, Morik et al. [37] propose a new algorithm that is able to take into account notions of amortized group fairness while still learning user preferences. This is a fundamentally different setting than what we consider in that we are not performing learning to rank and we do not want to fix a set of fairness criteria a priori into our recommendation algorithm; rather, we treat the recommendation algorithm itself as an input.

Morik et al. [37] investigate the problem of learning to rank over large item sets while ensuring fairness of merit-based guarantees to groups of item producers. Specifically, they adapt existing methods to ensure that the exposure is unbiased, e.g., that it is not subject to rich-get-richer dynamics, and fairness defined as exposure being proportional to merit. Both of these goals are built into the regularization of the learner. In essence, the goal is to learn user preferences while ensuring the above two desiderata. In contrast, our work factors out the recommendation methodology and we encapsulate the desired fairness definitions as separate agents rather than embedded in the learning algorithm.

Similarly to the RL settings described above, other recommendation systems contexts include session-based and streaming (sequential) recommendation. Wu et al. [59] investigate the setting of session-based recommender systems that are short, memory-less recommendation experiences with, e.g., non-logged-in users on a website. Like our work, they propose using the recommendation lists delivered over time to give the overall system some memory as to how fair it has been in the past, proposing a new time-decaying notion of exposure fairness. They also employ a post-processing concept for the overall recommendation: As they focus on session-based recommender systems, there is no notion of long-term user engagement. This is an important direction for potential future work, e.g., non-logged-in users are prevalent in many domains. Within the sequential recommender setting, Li et al. [33] propose a system for a (set of) users that incorporates feedback in learning both the preferences of the agent as well as a regularization term in the online learning algorithms to control for a type of interaction fairness per user. They use a deep learning framework work and a knowledge graph over the items to embed the user feedback and interaction and then use this to ensure that fairness of interaction is happening across protected item groups. Their model assumes fairness on a per-user (though dynamic) basis but more importantly that a large knowledge graph of item information is available, where we make no such assumptions and can define fairness per-user, per-item, and across user sets.

In early stages of this project, we presented an alpha version of the SCRUF-D architecture at a non-archival workshop Sonboli et al. [51]. The alpha version of SCRUF-D considered only a small set of fixed lotteries for allocation and only a single fairness agent and computed fairness metrics at fixed intervals rather than in a fully dynamic way. Through extensive work with Kiva to identify additional concerns and fairness definitions [49], we continued to iterate on the system. The version reported in this work incorporates the ability to dynamically compute allocations for each user arrival based on multiple agents and definitions of fairness, a more extensive set of fairness definitions, choice rules, re-rankers, recommendation models, and metrics.

Similarly to SCRUF-D and working with stakeholders, Ferraro et al. [23] propose a novel set of re-rankers for a music recommendation platform after a set of interview studies with users and producers of music on the system. After these studies, they find that gender fairness is a key issue for many artists, with real-world data they propose a re-ranking methodology that attempts to address this imbalance. The overall methods in this article mirrors ours in that a close examination of a real-world system, and interviews with stakeholders, give rise to a particular notion and measurement of fairness [49]. However, rather than a case study, we propose a more robust framework that treats the multistakeholder and contextual definitions of fairness as first–order concerns.

Finally, our recommendation allocation problem has some similarities with those found in computational advertising, where specific messages are matched with users in a personalized way [56, 60]. Because advertising is a paid service, these problems are typically addressed through mechanisms of monetary exchange, such as auctions. There is no counterpart to budgets or bids in our context, which means that solutions in this space do not readily translate to supporting fair recommendation [20, 61, 65].

Consider the following set of fairness agents and their associated fairness definitions, evaluations, and intervention/preference (re-ranker/weighting of items). We assume in this example that in all cases the agents’ compatibility functions follow the pattern described in Sonboli et al. [52] where the entropy of the user profile relative to the sensitive feature is calculated and users with high entropy are determined to be good targets for fairness-enhancing interventions. Note that in these examples, the definition of a fair outcome is different for each agent, demonstrating the range of fairness definitions that can be incorporated in our system as follows:

Consider the contents of Table 1. For the sake of example, we will assume these loans, characterized by the Region, Gender, Section, and Amount, constitute the set of loans available for recommendation.

For the sake of exposition, we posit two very simple mechanisms for allocation and choice. We will assume that our allocation mechanism is a single outcome lottery, e.g., a randomized allocation mechanism [8], wherein one agent will be chosen to participate in the choice mechanism, based on a random draw with probabilities based on the historic unfairness and user compatibility as measured by each agent.

We assume that the recommendation lists are of size 3 and the choice mechanism uses a weighted voting/score-based mechanism [7] using a weighted sum of 0.75 on the personalized results for the recommender system and 0.25 on the output of the allocated fairness agent.

At time \(t_1\), User \(u_1\) arrives at the system and the recommendation process is triggered. The user has previously supported small loans only in Central America and Middle East but has lent to a wide variety of sectors and genders.

For the sake of example, we will assume that the agents measure their prior history relative to their objectives as equally unfair at 0.5, except the Gender Parity agent, which starts out at parity and therefore returns a value of 1. However, the compatibility functions for \(f_A\) and \(f_L\) returns lower scores because of the user’s historical pattern of lending. This yields a lottery in which \(f_G\) has probability zero, \(f_A\) has a low probability, and \(f_H\) a higher one. The allocation mechanism chooses randomly, and we will assume that \(f_H\), the health-focused agent, is picked.

The recommender returns the following list of items and predicted ratings \([\lbrace v_6, 0.6\rbrace , \lbrace v_4, 0.5\rbrace , \lbrace v_5, 0.3\rbrace , \lbrace v_3, 0.3\rbrace ,\) \(\lbrace v_1, 0.0\rbrace , \lbrace v_2, 0.0\rbrace ]\). The \(f_H\) agent gives a score of 1 to the health-related loans \(v_2\) and \(v_5\) and 0 to all others.

The choice mechanism combines these scores as described above and returns the final recommendation list \([\lbrace v_5, 0.475\rbrace , \lbrace v_6, 0.45\rbrace , \lbrace v_4, 0.375\rbrace ]\). Note that the Health agent has successfully promoted its preferred item to the first position in the list.

For the sake of example, we assume that the agents’ evaluation functions are very sensitive. Therefore, when User \(u_2\) arrives, the results of the previous recommendations have caused the evaluations to shift such that the Health \(f_H\) and Large \(f_L\) agents are now satisfied (note that \(v_4\) is included in \(u_1\)’s list and it was a large loan), the Gender parity agent \(f_G\) is now at 0.9 (note that there is only one male loan in the database) but the Africa agent \(f_A\), which got nothing in \(u_1\)’s list, considers recent results to be unfair with a score of 0.25. We assume that \(u_2\) is similar to \(u_1\) in profile and therefore compatibility, but \(f_A\) has a much worse fairness score than \(f_G\), and therefore a high allocation probability. We will assume \(f_A\) is chosen.

Because this user has similar preferences to \(u_1\), they get the same recommendations: \([\lbrace v_6, 0.6\rbrace , \lbrace v_4, 0.5\rbrace , \lbrace v_5, 0.3\rbrace , \lbrace v_3, 0.3\rbrace ],\) \(\lbrace v_1, 0.0\rbrace , \lbrace v_2, 0.0\rbrace ]\). The \(f_A\) agents scores the two loans from Africa (\(v_1\) and \(v_2\)) at 1 and the others at 0.

So, after randomly breaking the tie between \(v_1\) and \(v_2\), the final recommendation list is \([\lbrace v_6, 0.45\rbrace , \lbrace v_4, 0.375\rbrace , \lbrace v_1, 0.25\rbrace ]\).

When User \(u_3\) arrives, all four agents find themselves scoring fairness at 1 over the evaluation window and so no agents are allocated. The results from the recommendation algorithm pass through the choice mechanism unchanged and are delivered to the user.

In this example, we see the interplay between users’ compatibility with agents and the computed fairness outcomes to allocate opportunities to pursue fairness among different agents over time.

We can think of a recommender system as a two-sided market in which the recommendation opportunities that arise from the arrival of a user \(u\in \mathcal {U}\) to the system, and each are allocated to a set of items \(v\in \mathcal {V}\) from the system’s catalog. This market has some similarities to various forms of online matching markets including food banks [2], kidney allocation [4, 35], and ride sharing [19] in that users have preferences over the items; however, in our case this preference is known only indirectly through either the prior interaction history or a recommendation function. Additionally, the items are not consumable or rivalrous. For example, a loan can be recommended to any number of users—it is not “used up” in the recommendation interaction.⁷ Also, users are not bound to the recommendations provided; in most online platforms including Kiva, there are multiple ways to find items, of which the recommender system is only one.

Once we have a collection of fairness agents we must solve two interrelated problems:

We assume a recommendation generation process that happens over a number of timesteps t as individual users arrive and recommendations are generated on demand. Users arrive at the system one at a time, receive recommendations, act on them (or not), and then depart. When a user arrives, a recommendation process produces a recommendation list \(\ell _s\) that represents the system’s best representation of the items of interest to that user, generated through whatever recommendation mechanism is available. We do not make any assumptions about this process, except that it is focused on the user and represents their preferences. A wide variety of recommendation techniques are well studied in the literature, including matrix factorization, neural embeddings, graph-based techniques, and others.

The step of determining which fairness concerns/agents will be active in responding to a given recommendation opportunity is the allocation phase of the process, the output of which is a set of non-negative weights \(\beta\), summing to one, over the set of fairness agents, indicating to what extent each fairness agent is considered to be allocated to the current opportunity [7].

Once the set of fairness agents have been allocated, they have the opportunity to participate in the next phase of the process, which is the choice phase. In this phase, all of the active (non-zero weighted) agents and their weights participate in producing a final list of recommendations for the user. We view the recommender system itself as being an agent that is always allocated and therefore always participates in this phase.

Figure 2 shows the allocation phase of this process. This is an online and dynamic allocation problem where we may consider many factors including the history of agent allocations so far, the generated lists from past interactions with users, and how fair the set of agents believes this history to be. As described in Section 5.2 below, agents take these histories and information about the current user profile and calculate two values: m, a measure of fairness relative to their agent-specific concern, and c, a measure of compatibility between the current context and the agent’s fairness concern. The allocation mechanism takes these metrics into account producing a probability distribution over the fairness agents that we call the agent allocation, which can be interpreted as weights in the choice stage or be used to select a single agent via a lottery, e.g., a randomized allocation scheme [8].

In the second phase, shown in Figure 3, the recommender system generates a list of options, considered to represent the user’s preferences. The fairness concerns generate their own preferences as well. These preferences may be global in character, i.e., preferences over all items, in which case they may be independent of what the recommender system produces; we call this a recommendation function below. Or, as indicated by the dashed line, these preferences may be scoped only over the items that the recommender system has generated; named a scoring function. In either case, the preference function of the fairness agent, like the one for the user, generates a list of items and scores. The choice mechanism combines these preferences of both the user and fairness agents, along with the allocation weights of the fairness agents, to arrive at a final recommendation list to be delivered to the user. The list itself, and possibly the interactions the user has with it, becomes a new addition to the choice history and the process continues for the next user. Because the output of the base recommender system \(\mathcal {R}_x\) is an input to this phase, we can think of the entire process as one of fairness-aware re-ranking.

As we have discussed, fairness concerns, derived from stakeholder consultation, are instantiated in the form of fairness agents, each having three capabilities, formalized in Table 3:

In our formalization of a recommendation system setting we have a set of n users \(\mathcal {U}= \lbrace u_1, \ldots u_n\rbrace\) and a set of m items \(\mathcal {V}= \lbrace v_1, \ldots , v_m\rbrace\). For each item \(v_i \in \mathcal {V}\) we have a k-dimensional feature vector \(\phi = \langle \phi _{1}, \ldots \phi _{k} \rangle\) over a set of categorical features \(\phi\), each with finite domain. Some of these features may be sensitive, e.g., they are associated with one or more fairness agent concerns, we denote this set as \(\phi ^{s}\). Without loss of generality, we assume that all elements in \(\mathcal {V}\) share the same set of features \(\phi\). Finally, we assume that each user is associated with a profile of attributes \(\omega = \langle \omega _{1}, \ldots \omega _{j} \rangle\), of which some also may be sensitive \(\omega ^{s} \subseteq \omega\), e.g., they are associated with one or more fairness agents.

As in a standard recommendation system we assume that we have (one or more) recommendation mechanisms that take a user profile \(\omega\) and a (set of) items v and produces a predicted rating \(\hat{r}\in \mathbb {R}_+\). We will often refer to a recommendation list, \(\ell = \langle \lbrace v_1, \hat{r}_1\rbrace , \ldots \lbrace v_m, \hat{r}_m\rbrace \rangle\), which is generated for user \(\omega\) by sorting according to \(\hat{r}\), i.e., \(sort(\mathcal {R}_x(\omega , \mathcal {V})) \rightarrow \ell\). Note that this produces a permutation (ranking) over the set of items for that user, i.e., a recommendation. As a practical matter, the recommendation results will almost always contain a subset of the total set of items, typically the head (prefix) of the permutation up to some cutoff number of items or score value. For ease of exposition we assume we are able to score all items in the database.

In the SCRUF-D architecture, fairness concerns are “embodied” as a set of d agents \(\mathcal {F}= \lbrace f_1, \ldots , f_d\rbrace\). For the agents to be able to evaluate their particular concerns, they take account of the current state of the system and voice their evaluation of how fairly the overall system is currently operating, their compatibility for the current recommendation opportunity, and their preference for how to make the outcomes more fair. Hence, each fairness agent \(i \in |\mathcal {F}|\) is described as a tuple, \(f_i = \lbrace m_i, c_i, \mathcal {R}_i\rbrace\) consisting of a fairness metric, \(m_i(\vec{L}, \vec{H}) \rightarrow [0,1]\), that takes a choice history \(\vec{L}\) and allocation history \(\vec{H}\) and produces a value in \([0,1]\) according to the agent’s evaluation of how fair recommendations so far have been; a compatibility metric, \(c_i(\omega) \rightarrow [0,1]\), that takes a particular user profile \(\omega\) and produces a value in \([0,1]\) for how compatible fairness agent i believes they are for user \(\omega\); and a ranking function, \(\mathcal {R}_i(\omega , v) \rightarrow \lbrace v, \hat{r}\rbrace\), that gives the fairness agent preferences/ranking over the set of items.

In the allocation phase (Figure 2), we must allocate a set of fairness agents to a recommendation opportunity. Formally, this is an allocation function, \(\mathcal {A}(\mathcal {F}, m_\mathcal {F}(\vec{L}, \vec{H}), c_\mathcal {F}(\omega)) \rightarrow \beta \in \mathbb {R}_{+}^{|\mathcal {F}|}\) that takes a set of fairness agents \(\mathcal {F}\), the agents’ fairness metric evaluations \(m_\mathcal {F}(\vec{L}, \vec{H}) = \lbrace m_1(\vec{L}, \vec{H}), \ldots , m_d(\vec{L}, \vec{H})\rbrace\), and the agents’ compatibility metric evaluations \(c_\mathcal {F}(\omega)\) and maps to an agent allocation \(\beta\), where \(\beta\) is a probability distribution over the agents \(\mathcal {F}\). The allocation function itself is allocating fairness agents to recommendation opportunities by considering both the fairness metric for each agent as well as each fairness agent’s estimation of their compatibility.

The allocation function can take many forms, e.g., it could be a simple function of which every agent voices the most unfairness in the recent history [51], or it could be a more complex function from social choice theory such as the probabilistic serial mechanism [6] or other fair division or allocation mechanisms. Note here that the allocation mechanisms is directly comparing the agent valuations of both the current system fairness and compatibility. Hence, we are implicitly assuming that the agent fairness evaluations are comparable. While this is a somewhat strong assumption, it is less strong than assuming that fairness and other metrics, e.g., utility or revenue, are comparable, as is common in the literature [66]. So, although we are assuming different normalized fairness values are comparable, we are only assuming that fairness is comparable with fairness, and not other aspects of the system. We explore options for the allocation function in our empirical experiments below. We track the outputs of this function as the allocation history, \(\vec{H}= \langle \beta ^1, \ldots , \beta ^t \rangle\), an ordered list of agent allocations \(\beta\) at time t.

In the second phase of the system (Figure 3), we take the set of allocated agents and combine their preferences (and weights) with those of the current user \(\omega\). To do this we define a choice function, \(\mathcal {C}(\ell , \beta , \ell _\mathcal {F}) \rightarrow \ell _\mathcal {C}\), as a function from a recommendation list \(\ell\), agent allocation \(\beta\), and fairness agent recommendation list(s) \(\ell _\mathcal {F}\) to a combined list \(\ell _\mathcal {C}\). Each of the fairness agents is able to express their preferences over the set of items for a particular user, \(\mathcal {R}_i(\omega , v) \rightarrow \lbrace v, \hat{r}\rbrace\), and we take this set of lists,\(\ell _\mathcal {F}= \lbrace \mathcal {R}_1(\omega , \mathcal {V}), \ldots , \mathcal {R}_i(\omega , \mathcal {V})\rbrace\), as input to the choice function that generates a final recommendation that is shown to the user, \(\ell _\mathcal {C}\).

We again leave this choice function unspecified as this formulation provides a large design space: We could use a simple voting rule, a simple additive utility function or something much more complicated like rankings over the set of all rankings [7]. Note that the choice function can use the agent allocation \(\beta\) as either a lottery to, e.g., select one agent to voice their fairness concerns, or as a weighting scheme. We investigate a range of choice functions in our experiments. For the fairness agents to be able to evaluate the status of the system we also track the choice history, \(\vec{L}= \langle \ell ^t, \ell ^t_\mathcal {F}, \ell ^t_C \rangle\), as an ordered list of user recommendation list \(\ell\), agent recommendation list(s) \(\ell _\mathcal {F}\), and choice function output lists \(\ell _\mathcal {C}\), indexed by timestep t.

We can expect that an agent associated with a fairness concern will typically have preferences that order items relative to a particular feature or features associated with that concern. Items more closely related to the sphere of concern will be ranked more highly and those unrelated, lower. However, this property means that agents associated with different concerns might have quite different rankings—the gender parity concern will rank women’s loans highly regardless of their geography, for example. Thus, we cannot assume that these preferences exhibit nice social choice theoretic properties like consistency or single-peakedness across the different agents [67].

As noted above, fairness agents may have preferences over disjoint sets of items or they may be constrained only to have preferences over the items produced by the recommender system for the given user. This second option corresponds to a commonly used re-ranking approach, where the personalization aspect of the system controls what items can be considered for recommendation and fairness considerations re-order the list [21]. If an agent can introduce any item into its preferences, then we may have the challenge in the choice phase of integrating items that are ranked by some agents but not others. Some practical work-arounds might include a constraint on the recommender system to always return a minimum number of items of interest to the allocated agents or a default score to assign to items not otherwise ranked.

Despite our fairness-oriented motivation, it should be clear that our architecture is sufficiently general that an agent could be designed that pushes the system to act in harmful and unfair ways rather than beneficial and fairness-enhancing ones. The system has no inherent representation of fairness and would not be able to detect such usage. Thus, the importance of the initial step of stakeholder consultation and the careful crafting of fairness concerns. Because fairness concerns are developed within a single organization and with beneficence in mind, we assume that we do not need to protect against adversarial behavior, such as collusion among agents or strategic manipulation of preferences. The fact that the agents are all “on the same team” allows us to avoid constraints and complexities that otherwise arise in multi-agent decision contexts.

The ability of an agent to address its associated fairness concern is non-deterministic. It is possible that the agent may be allocated to a particular user interaction, but its associated fairness metric may still fail to improve. One likely reason for this is the primacy of the personalization objective. Generally, we expect that the user’s interests will have the greatest weight in the final recommendations delivered. Otherwise, the system might have unacceptably low accuracy, and fail in its primary information access objective.

One design decision therefore is whether (and how) to track agent efficacy as part of the system history. If the agent’s efficacy is generally low, then opportunities to which it is suited become particularly valuable; they are the rare situations in which this fairness goal can be addressed. Another aspect of efficacy is that relationships among item characteristics may mean that a given agent, while targeted to a specific fairness concern, might have the effect of enhancing multiple dimensions of fairness at once. Consider a situation in which geographic concerns and sectoral concerns intersect. Promoting an under-served region might also promote an under-served economic sector. Thus, the empirically observed multidimensional impact of a fairness concern will need to be tracked to understand its efficacy.

Efficacy may also be a function of internal parameters of the agent itself. A separate learning mechanism could then be deployed to optimize these parameters on the basis of allocation, choice and user interaction outcomes.

Different SCRUF-D implementations may differ in what aspects of the context are known to the allocation and/or choice mechanisms. Our hope is that we can leverage social choice functions to limit the complexity of the information that must be passed to the allocation and/or choice mechanisms. However, if a sophisticated and dynamic representation of agent efficacy is required, then it may be necessary to implement a bandit-type mechanism to explore the space of allocation probabilities and/or agent parameters as discussed above. Recent research on multidimensional bandit learning suggests possible approaches here [36].

As we have shown, agent priority in the allocation phase may be a function of user interests, considering different users as different opportunities to pursue fairness goals. It may also be a function of the history of prior allocations, or the state of the fairness concerns relative to some fairness metric we are trying to optimize. As the efficacy consideration would indicate, merely tracking allocation frequency is probably insufficient and it is necessary to tie agent priority to the state of fairness. Allocation priority is also tied to efficacy as noted above. It may be necessary to compute expected fairness impact across all dimensions to optimize the allocation.

We plan to leverage aspects of social choice theory to help ameliorate some of these issues. There is a significant body of research on allocation and fair division mechanisms that provide a range of desirable normative properties including envy-freeness [17], e.g., the guarantee that one agent will not desire another agent’s allocation, Pareto optimally, e.g., that agents receive an allocation that is highly desirable according to their compatibility evaluations [6]. An important and exciting direction for research is understanding what allocation properties can be guaranteed for the SCRUF-D architecture overall depending on the allocation mechanism selected [7].

We note that in most practical settings the personalization goal of the system will be most important and therefore the preference of this agent will have topmost priority. It is always allocated and is not part of the allocation mechanism. Thus, we cannot assume that the preference lists of the agents that are input to the choice system are anonymous, a common assumption in the social choice literature on voting [7].

Depending on how the concept of agent/user compatibility is implemented, it may provide benefits to bossy users, those with very narrow majoritarian interests that do not allow for the support of the system’s fairness concerns. Those users get results that are maximally personalized and do not share in any of the potential accuracy losses associated with satisfying the system’s fairness objectives. Other, more tolerant users, bear these costs. A system may wish to ensure that all users contribute, at some minimal level, to the fairness goals. In social choice theory, a mechanism is said to be non-bossy if an agent cannot change the allocation without changing the allocation that they receive by modifying their preferences [43]. Some preliminary discussions of this problem specifically for fairness-aware recommendation appear in Farastu et al. [22].

We concentrate in this article and our work with Kiva generally on provider-side group fairness, that is characteristics of loans where protected groups can be distinguished. However, it is also possible to use the framework for other fairness requirements. On the provider side, an individual fairness concern is one that tracks individual item exposure as opposed to the group as a whole. It would have a more complex means of assessing preference over items and of assessing fairness state, but still fits within the framework.

Consumer-side fairness can also be implemented through use of the compatibility function associated with each agent. For example, the example of assigning risk appropriately based on user risk tolerance becomes a matter of having a risk reduction agent that reports higher compatibility for users with lower risk tolerance.

The agent-specific fairness metric allows each agent to calculate their current state of fairness given the user interactions that have occurred within the evaluation window. As we have noted above, SCRUF-D allows for a wide variety of metrics and makes no assumptions that agents have metrics with similar logic. For the experiments in this study, we have chosen to have uniform fairness metric across agents to more easily assess the impact of varying other platform parameters including the allocation and choice mechanisms.

For each agent defined on fairness concern relative to each dataset, we assign a target exposure value as noted above. That is, an agent with a 20% target exposure will return a value of 1.0 (perfect fairness) if, across all of the recommendation lists in the evaluation window, there is at least an average of 20% of items with its associated protected feature. More items do not result in a higher score, but fewer items would yield a value linearly scaled toward zero, as the value when no protected items have been recommended. We have set the targets artificially high to investigate how the system respond to this pressure to include more protected items.

As noted above, there is a wide variety of different allocation mechanisms that can be studied. For our purposes in this article, we are exploring mechanisms with widely differing logics to understand the implication of these choices for recommendation outcomes.

We examine four different choice mechanisms. In computational social choice, choice mechanisms are classically understood as integrating the preferences of multiple agents together to form a single societal preference [7].¹²

Rescore: The simplest mechanism is one in which each agent contributes a weighted score for each item and these scores are summed to determine the rank of items. Each fairness agent has a fixed score increment \(\delta\) that is added to all protected items, weighted by its allocation in the previous phase. This is combined with the scores computed by the recommendation algorithm.

Borda: Under the Borda mechanism [67], ranks are associated with scores and the original scores used to compute those ranks are ignored. The ranks across agents are summed and the result determines the final ranking.

Copeland: The Copeland mechanism calculates a win–loss record for each item considering all item–item pairs in a graph induced by the preferences. Item i scores one point over item j if the majority of allocated agents prefer i to j. We then sum these pairwise match-ups for each item i and order the list of items using these scores [42].

Ranked Pairs: The Ranked Pairs voting rule [55] computes the pairwise majority graph as described for Copeland but orders the resulting ranking by how much a particular item wins by, selecting these to create a complete ranking, skipping a pair if and only if it would induce a cycle in the aggregate ranking.

Each of these choice mechanisms implements a fundamentally different logic for aggregating preferences: score based, ordinal based, consistency based, and pairwise preference [67]. As we show in our results, choice mechanisms yield quite different accuracy/fairness tradeoffs.

While the agent weights in these mechanisms fixed by the allocation mechanism (and normalized to 1), the recommender weight is a parameter, which determines how much the recommender systems results are emphasized relative to the fairness agents. Under different conditions and mechanisms, different recommender weights may be optimal. We refer to this weight throughout as \(\lambda\).

As noted in Section 2, there are very few recommendation algorithms that allow for dynamic reranking with multiple fairness concerns at once. The Multinomial FA*IR method described in [62] is not practical for recommendation because of its time complexity. For these experiments, we use OFair [52] and MultiFR [58].

OFair is a multi-group fairness-aware recommendation reranking method based on the technique of maximum marginal relevance [15] for diversity enhancement in reranking. OFair seeks to enhance fairness for multiple groups by treating each sensitive aspect as a dimension of diversity and greedily building a recommendation list by adding items that enhance its diversity in this respect. OFair adds an additional consideration of personalization by weighting different features according to user compatibility, building on the work of Reference [34] using profile entropy relative to the sensitive features. OFair also has a \(\lambda\) parameter, controlling how much the recommender systems scores are weighted in the reranking process.

Multi-FR is described as a “multi-objective optimization framework for fairness-aware recommendation” [58]. Multi-FR models fairness constraints using a smoothed stochastic ranking policy and optimizes for fairness constraints using multiple gradient descent. The method finds multiple solutions along a theoretical Pareto frontier and chooses the best solution using a least misery strategy. Note that Multi-FR uses a batch-oriented strategy, attempting to address the fairness concerns over the entire set of recommendations at once. Still, Multi-FR is one of the few existing algorithms for fairness-aware recommendation that supports multiple fairness concerns, including provider-side and consumer-side constraints.

One key limitations of Multi-FR is that it represents provider-side fairness only in terms of mutually exclusive provider groups. SCRUF-D has no such limitation and supports intersecting fairness concerns. Because of this difference, in our experiments, we had to create a cross-product of all possible combinations of protected features to capture the multiple features in our datasets. Also, Multi-FR only supports a single type of fairness objective on the provider side: minimizing the difference between actual and ideal exposure of item categories. For the experiments below, target exposures were set as follows: Microlending: loan size/0.20, country/0.30; both features/0.026; MovieLens: women writer or director/0.09, non-English/0.25, both features/0.02. Multi-FR has its own method of balancing accuracy and fairness and so does not have a parameter controlling the balance between fairness and accuracy.

Batch-oriented processing as found in Multi-FR is a fairly common approach for fairness-aware recommendation [21] and it is true that recommendation results are often cached, so processing many users at once is a practical approach. However, it should be noted that a batch approach to fairness-aware recommendation does not guarantee fairness in the recommendations that are delivered. The system can guarantee that a good fairness/accuracy tradeoff is found across the recommendation lists that are processed in a given batch, but, these may not in fact be the recommendations that are delivered to users over any interval. The users compatible with a particular protected group (for example, those interested in foreign language movies) may not happen to show up very often. So, the careful balance between criteria achieved within the batch process may not be realized when the recommendations are delivered in practice: the recommendations that are fairness-enhancing may sit in the cache and never be output. One reason to prefer an on-demand approach to fairness enhancement is that it is responsive to fairness outcomes in the moment. Still, we have included Multi-FR as a comparator to indicate what batch-oriented algorithm can achieve by considering all the recommendations at once.

We evaluate ranking accuracy using normalized discounted cumulative gain (nDCG) and fairness using our (normalized) fair exposure relative to the target proportion set for each protected group. Note that the fairness metric as computed by each agent is different from the overall fairness computed over the experiment, for the simple reason that agents only look back over a fixed window in computing their fairness at each time point. We will use the notation \(\bar{m_i}\) to refer to the global fairness for agent i.

To derive a single score representing both the combined fairness of both agents and the disparity between them, we use the \(L_{1/2}\)-norm, which for our purposes is defined as

The factor of \(1/4\) is used to give the resulting value the same scale as the original fairness scores. If we consider a simple average \(\bar{m}^*\) across all the agents, then the \(L_{1/2}\) norm is maximized (and equal to \(\bar{m}j\)) if all of the fairness values are the same \(\bar{m}_i = \bar{m}^*\). A mix of lower and higher values with the same average will give a lower result.

A dynamic way to look at local fairness is to consider fairness regret over the course of the experiment. At each timestep, we calculate \(1 - m_i\), that is, the difference from perfect fairness as the agent defines it, and then sum these values over the course of the simulation. This is similar to the notion of regret in reinforcement learning but using fairness instead of utility. Cumulative fairness regret \(G_i\) for agent i is defined as

Figure 4 shows the results for the MovieLens data organized by choice mechanism and showing the results of adjusting \(\lambda\), the weight associated with the recommender system. In general, as one might expect, as the weight decreases, accuracy drops and fairness increases. We note that the pairwise Copeland and Ranked Pairs mechanisms prove difficult to tune when there is only a single agent being allocated. There is only a single value for all cases when \(\lambda \lt 1.0\). This is because these mechanisms depend only on rankings and so as long as the recommender outweighs the allocated agent, it wins all of the pairwise comparisons and when it does not, it loses all of them.

As was also seen in Reference [13], there exist some “win-win” regions with the Rescore mechanisms in conjunction with Weighted allocation. For small decreases in \(\lambda\), we see both fairness and accuracy increase at the same time, indicating that some reranking is actually beneficial to accuracy, even as measured in this off-line setting. In addition, the shallow slope of some of these curves suggests that implementers can increase fairness quite significantly over the baseline without too much loss of ranking accuracy. The other mechanisms have a steeper accuracy loss.

For the remainder of this discussion, we select the \(\lambda\) values where accuracy loss is less than or equal to 5% and consider what is the best fairness that can be achieved within this constraint. We do not have the ability to tune Multi-FR because of its design, which does Pareto optimization internally, so we report the results from this algorithm as designed.

Table 4 includes all of the findings across the different mechanisms. We include both the \(L_{1/2}\) norm and the average in the table. Where these are close in value, the agents are getting similar fairness outcomes. We can see that the women writers and directors fairness target is quite a bit more difficult to hit than the non-English target. There is a large difference already in the unreranked baseline and this carries through to the rerankers. Many of them are able to achieve and exceed the fairness target for the non-English feature, but none do better than 0.4 for the other feature. One exception is Multi-FR, which overshoots the fairness targets and ends up with very low accuracy. Both Least Fair and Weighted in conjunction with the Ranked Pairs mechanism do well in this respect. However, these mechanisms are not the best with respect to maintaining accuracy.

Figure 5 shows the accuracy vs. fairness results for all of the mechanisms and the baseline algorithms. We see some clustering by choice mechanism, except for the two pairwise algorithms, Copeland and Ranked Pairs. For these algorithms, the Lottery mechanism yields very poor ranking accuracy. For Copeland, this is also true of the Least Fair mechanism. The reason is that the Weighted mechanism is so different here is that it is bringing multiple agents to the choice mechanism in these cases. (The others only promote a single agent to the choice phase.) With these multiple agents in the mix, the recommender is dominant and so the agents are less effective at increasing fairness.

Except for the Rescore mechanism, the Lottery allocation data points are all dominated by other points along the Pareto frontier, which would seem to indicate a disadvantage to allocating only a single agent in the allocation phase. Theoretically, Lottery and Weighted allocations are the same in expectation, since the lottery is drawn from the same numerical distribution. However, that is not borne out in the results here. In the case of the Rescore mechanism, these two allocation mechanisms represent very different positions in the tradeoff space.

Overall, the Rescore and Ranked Pairs mechanisms occupy the dominant positions: Rescore at higher accuracy levels and Ranked Pairs at higher fairness levels. Somewhat surprisingly, the Least Fair allocation stakes out two spots on the Pareto frontier, even though it ignores user compatibility. It is generally lower in accuracy than Lottery or Weighted mechanisms when the same choice mechanism is applied although there are exceptions in the experiments.

Both OFair and Multi-FR are dominated by the SCRUF mechanisms at different points. In the case of Multi-FR, it appears to have inherent limitations on how much accuracy loss it is willing to entertain to increase fairness. We also note that its reranking decisions are made off-line in a batch mechanism and so it is not able to respond dynamically to fairness issues in the moment. OFair also has lower fairness and accuracy. Like the Weighted mechanism, it is trying to address all of the fairness concerns at once in each recommendation list.

Figure 6 shows the results for the Microlending dataset organized by choice mechanism adjusting \(\lambda\), the weight associated with the recommender system. The Microlending dataset turns out to be quite different from MovieLens for several reasons. One is that the fairness targets are much easier to achieve. As can be seen in Figure 6, almost all of the mechanisms are able to achieve fairness very close to 1.0. We also see fairly noisy behavior across the different \(\lambda\) values rather than the smoother accuracy/fairness tradeoff seen in the MovieLens data. We believe that this is a side-effect of the easier fairness target: There is not much trading off that the system needs to perform.

Table 5 shows the complete results and we note that the mechanisms are finding identical solutions in many cases, with the same fairness and accuracy values. This suggests that the original results contain only a limited number of protected items in each list and so there is only so much room for the reranker to alter the results. We believe that part of the reason for this effect is our choice of biased matrix factorization as our base recommendation algorithm. This algorithm is known to suffer from popularity bias and be limited in the range of items that it recommends [29]. In our future work, we will examine alternate base algorithms with better diversity.

The results for the Microlending data are summarized in the scatterplot in Figure 7. There only five points representing the 14 experimental conditions. OFair and Multi-FR are omitted, because they are far from the optimal tradeoff region (although we know from the table that Multi-FR has quite good accuracy). All four of the Least Fair conditions have the same values at the lower right, as do many of the Lottery conditions, except for Rescore.

With these limited data, it is hard to draw too many conclusions. Unlike in the MovieLens case, Least Fair occupies only the most extreme lower accuracy condition in these data. Rescore still seems to be a good strategy and Ranked Pairs with the Weighted allocation is still on the Pareto frontier as it was in the MovieLens case.

The compatibility of users with diverse sensitive features turns out to be highly correlated in our real-world datasets, which is perhaps not surprising, but it makes it difficult to explore dynamic scenarios simulating the arrival of disparate types of users at different times. For this reason, we used the Synthetic dataset described above in simulated experiments of user arrivals to examine how the balance between agents is achieved over time. With this synthetic dataset, we do not have ground-truth user preferences and so we evaluate recommendation accuracy only relative to the original rankings in the simulated data.

In Figure 8, we look closely at the cumulative fairness regret for the different allocation mechanisms. We keep the choice mechanism fixed (Borda) to isolate the impact of allocation on agent outcomes. Recall that the arrival of users is segmented so that users compatible with Agent 2 arrive as the first 500, followed by another 500 compatible with Agent 1, and then a third 500 user segment without strong compatibility to either agent. Note that the y-axis on these plots has a log scale.

For the Baseline algorithm, without reranking, the impact of the different segments can be seen in the flattening of the Agent 1 curve for users 500–1000. These users are compatible with Agent 1 and already have some of these protected items in their recommendation lists. The regret ends up quite high for both agents.

The OFair algorithm does not fare much better. It is trying to satisfy all of the fairness constraints at once. While its fairness regret is lower, especially for Agent 2, it is still quite high. The other allocation mechanisms fare much better, maintaining 10\(\times\) or greater improvement in regret over the course of the experiment. The Lottery and Weighted mechanisms are quite similar to each other, with the Weighted mechanism doing slightly better for Agent 2. By not trying (as hard) to satisfy Agent 1 when its compatible users are rare, the system is able to achieve better fairness for both agents.

The Least Fair mechanism seems even better still for both agents. However, there is more to the story. By ignoring compatibility, this mechanism cannot achieve ranking accuracy as high as the others as we have seen in our prior studies. For the synthetic data, we do not have ground-truth user data, so we represent the fairness/accuracy tradeoff by computing nDCG considering the input recommendations to be ground truth. This simulated nDCG measures how much reranking has occurred under each algorithm. These values are shown in Table 6, which shows the loss of accuracy incurred by each algorithm together with the overall fairness values. Differences between the algorithms are small (except for OFair) and we see that the Least Fair mechanism is lowest on accuracy.