Adjusted Count Quantification Learning on Graphs

Abstract

ql is the task of predicting the label distribution of a set of instances. We study this problem in the context of graph-structured data, where the instances are vertices. Previously, this problem has only been addressed via node clustering methods. In this paper, we extend the popular acc method to graphs. We show that the pps assumption upon which acc relies is often not fulfilled and propose two novel graph quantification techniques: sis makes acc applicable in graph domains with covariate shift. nacc improves quantification in the presence of non-homophilic edges. We show the effectiveness of our techniques on multiple graph quantification tasks.

1 Introduction

We consider the task of ql on graph-structured data. This term has first been coined by Forman [2005, 2006, 2008] and is used to describe the task of estimating label prevalences via supervised learning. A ql method receives a set of training instances with known labels which is used to train a quantifier. The quantifier is then used to predict the label distribution of a set of test instances. Unlike standard instance-wise classification, ql does not concern itself with predicting an accurate label for each test instance but rather with predicting the overall prevalence of each label across all instances. ql can thus be seen as a dataset-level prediction task, where a single prediction is made for a population of instances.

Quantification problems naturally arise in polling and surveying, where the goal is to estimate the proportion of a population that has a certain property or holds a certain opinion. Examples include estimating the proportion of voters that support a certain political party or the proportion of customers that are satisfied with a product. Similarly, ql can be applied to epidemiology or ecological modelling to estimate the prevalence of diseases or species in a given population. We refer to Esuli et al. [2023] for a comprehensive overview of the applications of quantification.

Typically, ql is studied in the context of tabular data, where each instance $x\in\mathcal{X}=\mathbb{R}^{d}$ is represented by a feature vector. In this setting, instances are assumed to be independent, i.e., the label distribution $p(y\mid x)$ is fully determined by the instance $x$. However, in many real-world applications, this independence assumption does not hold. Consider the example of estimating the proportion of voters supporting a certain party. Assume we have access to a social network where each node represents a voter and each edge represents a social connection. In this case, the label distribution of a voter, i.e., their political preferences, may depend not only on their own features but also on the features of their social connections. Incorporating this relational information into the quantification process can lead to more accurate estimates.

Generally speaking, ql methods can be divided into two categories: aggregative and non-aggregative. Aggregative quantifiers rely upon an instance-wise label estimator, i.e., a regular classifier; the instance-level label estimates are then aggregated to obtain dataset-level label prevalence estimates. Non-aggregative quantifiers, on the other hand, directly estimate dataset-level label prevalences without first predicting labels for each instance. In this paper, we focus on aggregative quantification methods, which are more common and have been studied more extensively. An intuitively plausible aggregative method is to simply estimate the prevalence of a label as the fraction of test instances that are predicted to belong to that label by the classifier. This method is known as cc and, given a perfect classifier, it will yield perfect quantification results. However, in practice, classifiers are not perfect and even good, but not perfect, classifiers can lead to poor quantification results. Conversely, even a bad classifier can yield good quantification results. The reason for this disconnect is that the optimization goals of classification and quantification are misaligned. More specifically, while a good binary classifier should minimize the total number of misclassifications, i.e., $(\mathrm{FP}+\mathrm{FN})$, a good binary quantifier should minimize $\left|\mathrm{FP}-\mathrm{FN}\right|$. If $\mathrm{FP}=\mathrm{FN}$, even a classifier with a high misclassification rate will yield perfect quantification results.

This misalignment is commonly addressed by the family of acc methods, which use an estimate of the classifier’s confusion matrix to adjust the predicted label prevalences [Vucetic and Obradovic, 2001, Saerens et al., 2002, Forman, 2005]. acc has been shown to estimate the true test label prevalences in expectation if the so-called pps assumption holds, i.e., if the class conditional training distributions $p(x\mid y)$ equals the class conditional test distribution $q(x\mid y)$ [Tasche, 2017].

In the following, we investigate the validity of the pps assumption and thereby of acc in the context of graph-structured data. In Section 2, we begin with a brief formal description of the quantification problem, the pps assumption and how it is used in acc. Section 3 describes two novel structure-aware aggregative quantification methods for graphs: *sis and *nacc. To our knowledge, this is the first work on graph quantification learning using a node classifier. In Section 4, the proposed methods are evaluated on a series of node classification datasets under different shift assumptions. Last, we conclude with a brief outlook in Section 5.

2 Quantification Learning

Let $\mathcal{X}$ denote the instance space and $\mathcal{Y}=\{1,\dots,K\}$ the (finite) label space. In ql we assume to be given a training set of labeled instances $\mathcal{D}_{L}\subseteq\mathcal{X}\times\mathcal{Y}$ drawn from a distribution $P(x,y)$ with corresponding density $p$. Additionally, there is a set of labeled instances $\mathcal{D}_{U}\subseteq\mathcal{X}\times\mathcal{Y}$ drawn from a test distribution $Q(x,y)$ with corresponding density $q$. The goal of ql is to estimate $Q(Y=i)$ for all $i\in\mathcal{Y}$ given $\mathcal{D}_{L}$ and $\mathcal{X}_{U}\coloneqq\{x\mid(x,y)\in\mathcal{D}_{U}\}$. If $P=Q$, i.e., if the training and test data are drawn from the same distribution, the quantification problem is trivially solved via a maximum likelihood estimate of the label distribution on $\mathcal{D}_{L}$:

$$\hat{Q}^{\mathrm{MLPE}}({Y=i})\coloneqq\frac{1}{|\mathcal{D}_{L}|}\smashoperator[r]{\sum_{(x,y)\in\mathcal{D}_{L}}^{}}\mathds{1}[{y=i}]$$ (1)

where $\mathds{1}[\cdot]$ denotes the indicator function. This mlpe approach [Barranquero et al., 2013, Esuli et al., 2023] is akin to the majority classifier in classification in the sense that it predicts the most likely distribution in the absence of test data $\mathcal{X}_{U}$. However, if the training and test data are not identically distributed, the quantification problem becomes more challenging. A quantification approach has to account for the distribution shift between $P$ and $Q$ to provide accurate estimates of $Q(Y)$. Depending on the nature of this distribution shift, different quantification methods may be more or less suitable.

2.1 Types of Distribution Shift

If the train and test distributions differ, one should ask whether learning from the training data is still feasible. Certainly, if $P$ and $Q$ are completely unrelated, any information learned from $\mathcal{D}_{L}$ is useless for predicting $Q(Y)$. Quantification approaches therefore typically assume that $P$ and $Q$ are related in some way. The applicability of a quantification method then depends on whether those assumptions hold true for the given problem. First, note that $q$ can be expressed as

$$q(x,y)=q(y\mid x)q(x)=q(x\mid y)q(y)\ .$$

By fixing one of the factors in the two right-hand terms, we obtain three types of distribution shifts [Esuli et al., 2023]:

1. Concept Shift: The conditional label distribution changes, but the distribution of the instances remains the same, i.e., $q(y\mid x)\neq p(y\mid x)$, while $q(x)=p(x)$. This type of shift, also referred to as concept drift, can occur in domains with classes that are defined relative to some frame of reference. For example, consider the task of predicting the prevalence of local vs. world news articles in a newspaper. While the distribution of news articles may remain the same between training and test, the definition of what constitutes local or world news depends on the location of the newspaper.

2. Covariate Shift: The distribution of the instances changes, but the conditional label distribution remains the same, i.e., $q(x)\neq p(x)$, while $q(y\mid x)=p(y\mid x)$. This is common in domain adaptation, where the training and test data are drawn from different but related domains. For example, assume the task is to predict the prevalence of a certain sentiment or opinion in social media posts. The training data may be drawn from one social media platform, while the test data is drawn from another. Given a post $x$, the probability of it expressing a certain sentiment $y$ is likely the same on both platforms, but the distribution of posts may differ.

3. Prior Probability Shift: The label distribution changes, but not the class-conditional instance distribution, i.e., $q(y)\neq p(y)$, while $q(x\mid y)=p(x\mid y)$. Similar to covariate shift, pps occurs between domains that share the same label concepts. For example, consider the task of predicting the percentage of a population that has a certain disease. The training data may come from a case-control study consisting of an equal amount of healthy and infected individuals, while the test data is drawn from the general population. Given $y\in\{\mathit{infected},\mathit{healthy}\}$, the feature distribution of an individual $x$ should be the same between training and test, whereas the prevalence of the disease will likely not be.

We do not consider the case where $q(y)=p(y)$, as this would imply that the label distribution remains unchanged, in which case the quantification problem is trivially solved by mlpe. Note that the difference between covariate shift and pps is subtle. Whether it is $p(x)$ or $p(y)$ that changes between training and test is mostly a matter of the assumed causal relation between instances and labels, i.e., whether it is in the direction $\mathcal{X}\to\mathcal{Y}$ or $\mathcal{Y}\to\mathcal{X}$ [Fawcett and Flach, 2005, Schölkopf et al., 2012, Kull and Flach, 2014]. In ql, pps is commonly assumed, as there are many $\mathcal{Y}\to\mathcal{X}$ domains in which this is reasonable [González et al., 2024]. Generally speaking, quantification under concept or covariate shift is more challenging and often requires additional assumptions or domain knowledge. We will get back to the question of which shift assumptions are appropriate for a given domain in Section 3.

2.2 Adjusted Count

We will now describe the acc method, a popular approach to quantification under pps [Forman, 2005]. As mentioned in the introduction, the naïve cc method estimates the prevalence of a label as the fraction of test instances that are predicted to belong to that label by a classifier $h:\mathcal{X}\to\mathcal{Y}$:

$$\hat{Q}^{\mathrm{CC}}(Y=i)\coloneqq\frac{1}{|\mathcal{X}_{U}|}\smashoperator[r]{\sum_{x\in\mathcal{X}_{U}}^{}}\mathds{1}[h(x)=i]\ .$$ (2)

Since $h$ is trained on data drawn from $P$, the estimated propensity scores $\hat{Q}^{\mathrm{CC}}(Y)$ will be biased towards $P(Y)$. acc removes this bias by adjusting the predicted label prevalences based on an estimate of the classifier’s confusion matrix. To understand acc, note that the pps assumptios implies that

$$Q(\hat{Y}=j) =\sum_{i=1}^{K}Q(\hat{Y}=j\mid Y=i)\cdot Q(Y=i)$$

$$=\sum_{i=1}^{K}P(\hat{Y}=j\mid Y=i)\cdot Q(Y=i)$$ (3)

for all $j\in\mathcal{Y}$, with $\hat{Y}=h(X)$. Given $\mathcal{D}_{L}$ and $\mathcal{X}_{U}$, we can obtain unbiased estimates of, both, $Q(\hat{Y})$ and $P(\hat{Y}\mid Y)$:

$$\hat{Q}(\hat{Y}=j) =\hat{Q}^{\mathrm{CC}}(Y=j)\ ,$$

$$\hat{P}(\hat{Y}=j\mid Y=i) =\frac{\smashoperator[r]{\sum_{(x,y)\in\mathcal{D}_{L}}^{}}\mathds{1}[h(x)=j\land y=i]}{|\{(x,y)\in\mathcal{D}_{L}\mid y=i\}|}\ .$$ (4)

Plugging these estimates into Eq. 3 yields a system of equations which can be solved to obtain estimates of $Q(Y)$ [Saerens et al., 2002]. Let $\hat{\mathbf{C}}\in{[0,1]}^{K\times K}$ be the estimated confusion matrix of $h$ on $P$, i.e., $\hat{C}_{j,i}=\hat{P}(\hat{Y}=j\mid Y=i)$. Then, the acc estimates of $Q(Y)$ are given by

$$\hat{Q}^{\mathrm{ACC}}(Y)=\hat{\mathbf{C}}^{-1}\cdot\hat{Q}(\hat{Y})\ .$$ (5)

While the binary version of acc goes back at least to Gart and Buck [1966], it was first described as a quantification method by Vucetic and Obradovic [2001]. Tasche [2017] showed that acc is an unbiased estimator of the true test label prevalences if the pps assumption holds.

Note that there are two practical problems with Eq. 5: First, if $\mathbf{C}$ is not invertible, there might be no or multiple solutions for $\hat{Q}^{\mathrm{ACC}}(Y)$. Second, the adjusted label prevalences may not be a valid distribution over $\mathcal{Y}$, i.e., they could lie outside $[0,1]$ or not sum to one. Possible reasons for this are that the pps assumption might not be (fully) satisfied or simply that the estimates $\hat{\mathbf{C}}$ and $\hat{Q}(\hat{Y})$ are noisy, e.g., due to small sample sizes. A number of solutions to these problems have been proposed in the literature, including clipping and rescaling the estimates [Forman, 2008], adjusting the confusion matrix [Lipton et al., 2018], using the pseudo-inverse of $\mathbf{C}$ or replacing the system of equations with a constrained optimization problem [Bunse, 2022]. In this work, we will use the latter approach, i.e., constrained optimization, to solve Eq. 5:

$$\hat{Q}^{\mathrm{ACC}}(Y)=\operatorname*{arg\,min}_{\mathbf{q}\in\Delta_{K}}\left\|\hat{\mathbf{C}}\cdot\mathbf{q}-\hat{Q}(\hat{Y})\right\|_{2}^{2}\ ,$$ (6)

where $\Delta_{K}$ denotes the unit $(K-1)$-simplex. This problem can be solved numerically, e.g., using a (quasi-)Newtonian method such as slsqp. Bunse [2022] has shown that this approach is a sensible default choice, as it generally performs well in practice.

In addition to the cc and acc methods described above, which use a hard classifier $h:\mathcal{X}\to\mathcal{Y}$, one can also use a probabilistic classifier $h_{s}:\mathcal{X}\to\Delta_{K}$ [Bella et al., 2010]. Analogous to cc, pcc is defined as

$$\hat{Q}^{\mathrm{PCC}}(Y=i)\coloneqq\frac{1}{|\mathcal{X}_{U}|}\smashoperator[r]{\sum_{x\in\mathcal{X}_{U}}^{}}h_{s}(x)_{i}\ .$$ (7)

Likewise, pacc estimates $Q(\hat{Y})$ and $P(\hat{Y}\mid Y)$ using predicted label probabilities:

$$\hat{Q}(\hat{Y}=j) =\hat{Q}^{\mathrm{PCC}}(Y=j)\ , \hat{P}(\hat{Y}=j\mid Y=i) =\frac{\smashoperator[r]{\sum_{(x,y)\in\mathcal{D}_{L}}^{}}h_{s}(x)_{j}\cdot\mathds{1}[y=i]}{|\{(x,y)\in\mathcal{D}_{L}\mid y=i\}|}\ .$$ (8)

The motivation for using a soft classifier instead of a hard one is that predicted label probabilities can be more informative than hard labels. Whether this is truly the case is problem dependent and depends on the quality of the predicted probabilities.

3 Graph Quantification Learning

We now turn to the problem of ql on graph-structured data. In Section 2, we assumed that the instances in $\mathcal{D}_{L}$ and $\mathcal{D}_{U}$ are i.i.d. wrt. $P$ and $Q$ respectively. This assumption does not hold for graph-structured data, where the instances are the vertices of a graph and the labels are associated with the vertices. More specifically, let $G=(V,E)$ be a graph with vertex set $V$ and edge set $E\subseteq V\times V$. Each vertex $v_{i}\in V$ is associated with a feature vector $x_{i}\in\mathcal{X}$ and a label $y_{i}\in\mathcal{Y}$. We use $\mathcal{N}(v_{i})=\{v_{j}\mid(v_{i},v_{j})\in E\}$ to denote the set of neighbors of $v_{i}$. The edges in $G$ are used to encode homophily between vertices, i.e., similar vertices are more likely to be connected. Formally, an edge $(v_{i},v_{j})\in E$ should indicate that $P(y_{i}=y_{j})\geq\varepsilon$, with $\varepsilon$ being either a graph-specific constant or a function of an edge weight $w_{i,j}\in\mathbb{R}$. Since homophily is symmetric by definition, $G$ is undirected, i.e., $(v_{i},v_{j})\in E\Leftrightarrow(v_{j},v_{i})\in E$. Such homophilic graphs are commonly used to represent social networks, citation networks, co-purchase graphs or the World Wide Web.

Figure 1: The Amazon Photos co-purchase graph. Colors indicate vertex labels ($K=8$). The highlighted vertices are misclassifications by an *appnp classifier.

Figure 1 shows one such graph, namely the Amazon Photos co-purchase graph [Shchur et al., 2019], where vertices represent products, edges indicate that two products are frequently bought together and labels represent product categories. Due to homophily, the product categories form separate densely connected clusters, while cross-category edges are sparse.

Analogous to the tabular case, in gql we are given a training set of labeled vertices $\mathcal{D}_{L}$ drawn from a distribution $P$ and our goal is to estimate the label distribution of the vertices in a test set $\mathcal{D}_{U}$ drawn from a distribution $Q$. Given some vertex classifier $h_{G}:V\to\mathcal{Y}$, the gql problem is, in principle, amenable to standard aggregative quantification methods, such as acc or pacc. As discussed in Section 2.2, those adjusted count methods assume pps, which in turn assumes a $\mathcal{Y}\to\mathcal{X}$ domain. This means that both the training and the test data is assumed to be generated by sampling from some fixed distribution $p(x\mid y)$ for all $y\in\mathcal{Y}$. We argue that this is oftentimes not realistic for graph-structured data.

Consider the example of estimating the proportion of users holding a certain opinion. Here, the training data $\mathcal{D}_{L}$ may come from a social network where a (non-representative) local subset of users was sampled. The test data $\mathcal{D}_{U}$, on the other hand, may come from the entire social network or possibly some local subcluster of interest. In this setting, it is the instance distribution $p(x)$ that changes, while $p(y\mid x)$ remains fixed, i.e., covariate shift. More generally, a sampling process that is structure-dependent, e.g., by sampling local training or test neighborhoods, has covariate shift, not pps. We will now discuss how such structural biases can be accounted for in the quantification process.

3.1 Structural Importance Sampling

acc depends on being able to estimate the test confusion matrix $\mathbf{C}$ from training data, with $C_{j,i}\coloneqq Q(\hat{Y}=j\mid Y=i)$. As described, estimating $\mathbf{C}$ is trivial under pps. We will now introduce sis, a novel approach to gql under covariate shift. First, note that $\mathbf{C}_{j,i}$ can be expressed as

$$C_{j,i} =\smashoperator[]{\sum_{v\in V}^{}}\mathds{1}[h_{G}(v)=j]\cdot Q(v\mid Y=i)$$ (9)

$$=\smashoperator[]{\sum_{v\in V}^{}}\mathds{1}[h_{G}(v)=j]\underbrace{\frac{Q(v\mid Y=i)}{P(v\mid Y=i)}}_{=\rho(v\mid i)}\cdot P(v\mid Y=i)\ .$$

Using the covariate shift assumption, we can rewrite $\rho$ as

$$\rho(v\mid i) =\frac{Q(Y=i\mid v)\cdot Q(v)\cdot P(Y=i)}{P(Y=i\mid v)\cdot P(v)\cdot Q(y=i)}$$

$$=\frac{Q(v)}{P(v)}\cdot\frac{P(Y=i)}{Q(Y=i)}=\rho(v)\cdot\rho(i)^{-1}\ .$$ (10)

Thus, $\mathbf{C}$ can be obtained by reweighting the vertices:

$$C_{j,i} =\rho(i)^{-1}\smashoperator[]{\sum_{v\in V}^{}}\mathds{1}[h_{G}(v)=j]\cdot\rho(v)\cdot P(v\mid Y=i)$$

$$=\frac{\rho(i)^{-1}\smashoperator[]{\sum_{v\in V}^{}}\mathds{1}[h_{G}(v)=j]\cdot\rho(v)\cdot P(v\mid Y=i)}{\rho(i)^{-1}\smashoperator[]{\sum_{v\in V}^{}}\rho(v)\cdot P(v\mid Y=i)}$$

$$=\frac{\mathbb{E}_{v\sim P(\cdot\mid i)}\left[\mathds{1}[h_{G}(v)=j]\cdot\rho(v)\right]}{\mathbb{E}_{v\sim P(\cdot\mid i)}\left[\rho(v)\right]}\ .$$ (11)

Given $\mathcal{D}_{L}$, we can obtain an unbiased estimate of $C_{j,i}$:

$$\hat{C}_{j,i}=\frac{\smashoperator[r]{\sum_{(v,y)\in\mathcal{D}_{L}}^{}}\mathds{1}[h_{G}(v)=j\land y=i]\cdot\rho(v)}{\smashoperator[]{\sum_{(v,y)\in\mathcal{D}_{L}}^{}}\rho(v)\cdot\mathds{1}[y=i]}\ .$$ (12)

Note that this is essentially a weighted version of Eq. 4.

The problem with this formulation is that it requires $\rho(v)=\frac{Q(v)}{P(v)}$, which cannot be computed since $Q(v)$ is unknown. We do however have access to $\mathcal{X}_{U}$, which is sampled from $Q$. Using a suitable vertex kernel $k:V\times V\to\mathbb{R}$, we can thus estimate $\rho(v)$ via kernel density estimation:

$$\rho(v)\mathchoice{\mathrel{\vbox{ \offinterlineskip\halign{\hfil$#$\cr\displaystyle\propto\cr\kern 2.0pt\cr\displaystyle\sim\cr\kern-2.0pt\cr}}}}{\mathrel{\vbox{ \offinterlineskip\halign{\hfil$#$\cr\textstyle\propto\cr\kern 2.0pt\cr\textstyle\sim\cr\kern-2.0pt\cr}}}}{\mathrel{\vbox{ \offinterlineskip\halign{\hfil$#$\cr\scriptstyle\propto\cr\kern 2.0pt\cr\scriptstyle\sim\cr\kern-2.0pt\cr}}}}{\mathrel{\vbox{ \offinterlineskip\halign{\hfil$#$\cr\scriptscriptstyle\propto\cr\kern 2.0pt\cr\scriptscriptstyle\sim\cr\kern-2.0pt\cr}}}}\hat{\rho}(v)\coloneqq\frac{1}{|\mathcal{X}_{U}|}\smashoperator[r]{\sum_{v^{\prime}\in\mathcal{X}_{U}}^{}}k(v,v^{\prime})\ .$$ (21)

The suitability of the kernel $k$ depends on the nature of the test distribution $Q$. For example, if $Q(v\mid y)$ is uniform for all $y\in\mathcal{Y}$, i.e., the sampling is structure agnostic, the constant kernel $k_{1}(v,v^{\prime})=1$ would be appropriate. Note that Eq. 12 simplifies to standard acc in this case, as structure agnostic test sampling implies pps.

If the test nodes are sampled via a randomized bfs, a sp kernel, e.g.,

$$k_{\mathrm{SP}}(v,v^{\prime})=\exp(-\lambda\cdot d_{\mathrm{SP}}(v,v^{\prime}))\ ,$$ (22)

might be more appropriate, where $d_{\mathrm{SP}}(v,v^{\prime})$ is the length of the shortest path length between $v$ and $v^{\prime}$ and $\lambda>0$ a tunable hyperparameter. Similarly, for rw sampling, one can use a kernel based on the ppr algorithm [Page et al., 1999]:

$$k_{\mathrm{PPR}}(v,v^{\prime})=\Pi^{L}_{v^{\prime},v}\ ,$$ (23)

where $\Pi^{L}_{v^{\prime},v}$ denotes the probability that a random walk of length $L$ starting at $v^{\prime}$ ends at $v$. In general, the kernel should be treated as a hyperparameter which has to be tuned to the specific problem at hand.

To summarize, sis enables graph quantification under covariate shift by estimating the test confusion matrix $\mathbf{C}$ using a kernel density estimate of the test instance distribution $Q$. Using this estimate, the adjusted label prevalences can be computed using Eq. 6.

3.2 Neighborhood-aware Adjusted Count

In the previous section, we described how gql can be performed even under covariate shift. In addition to the nature of the distribution shift, there is another aspect to consider when applying acc: Class identifiability. Consider a classifier that is unable to distinguish between two classes $i$ and $j$, i.e., it predicts the same label for both. In this case, $\mathbf{C}_{:,i}=\mathbf{C}_{:,j}$ and thus there is no unique solution for Eq. 5. This can lead to poor quantification results if the prediction vector $Q(\hat{Y})$ has a large overlap with, both, $\mathbf{C}_{:,i}$ and $\mathbf{C}_{:,j}$, since any distribution of probability mass between both classes may then be returned. To address this issue, we propose nacc, which uses the neighborhood structure of the graph to improve class identifiability.

First, note that Eq. 6 can be understood as finding a mixture of the columns of $\mathbf{C}$ that best approximates $Q(\hat{Y})$. In the case of collinear columns, this mixture is not unique. A simple way to break such symmetries is to use the neighborhood structure of the graph:

$$Q(\hat{Y}_{\mathcal{N}}=(j,k)) =\sum_{i=1}^{K}Q(\hat{Y}_{\mathcal{N}}=(j,k)\mid Y=i)\cdot Q(Y=i),$$

where $\hat{Y}_{\mathcal{N}}$ is a RV representing a tuple of the predicted label of a vertex and the majority predicted label of its neighbors. Using this decomposition of $Q(\hat{Y}_{\mathcal{N}})$, $Q(Y)$ can be estimated using acc, and possibly sis, with the only difference being that the confusion matrix estimate is now of shape $K^{2}\times K$. Intuitively, this approach uses information on the presence of absence homophily to improve class identifiability.

Consider Fig. 1, where a vertex is highlighted if it is misclassified by an appnp classifier [Gasteiger et al., 2018]. Note that the vertices with label 7 (dark green) are often confused with vertices of label 1 (blue) or 6 (orange) because there are many non-homophilic edges between those classes. Using acc, this would imply that the row vectors of labels 7, 1 and 6 are collinear, i.e., $C_{:,7}\approx\alpha\cdot C_{:,1}+(1-\alpha)\cdot C_{:,6}$ for some $\alpha\in[0,1]$. Using the neighborhood structure, nacc can break this symmetry. For labels 1 and 6, the majority of predicted neighbors will nearly always be of the same label due to homophily, whereas for label 7, both, $\hat{Y}_{\mathcal{N}}=(1,6)$ and $\hat{Y}_{\mathcal{N}}=(6,1)$ are common. With this information, nacc is able to distinguish the confusion profile of label 7 from those of labels 1 and 6.

In principle, one could extend nacc to use even more neighborhood information, e.g., by considering the majority label of the neighbors of neighbors or by considering the second-most predicted neighboring label. However, given a finite training set $\mathcal{D}_{L}$, by making the confusion profiles for fine-grained, the confusion estimate $\hat{\mathbf{C}}$ will become noisier, counteracting the potential gains of additional information. We found that using the 1-hop majority label is a good trade-off between class identifiability and confusion estimate noise.

4 Evaluation

We assess the performance of sis and nacc on a series of graph quantification tasks using, both, pps and covariate shift. The quantification methods are applied to the predictions of multiple node classifiers. As a baseline we compare our proposed gql methods with mlpe, (P)CC and (P)ACC.

4.1 Experimental Setup

Quantification Metrics

There is a large number of metrics to evaluate quantification methods [Esuli et al., 2023]. We use the following three: The ae is one of the most commonly used metrics in quantification. It is defined as

$$\mathrm{AE}(q,\hat{q})=\frac{1}{K}\sum_{i=1}^{K}|q_{i}-\hat{q}_{i}|\ .$$ (24)

The rae [González-Castro et al., 2013] is a reweighted version of the ae that penalizes deviations from labels with low prevalence more heavily:

$$\mathrm{RAE}(q,\hat{q})=\frac{1}{K}\sum_{i=1}^{K}\frac{|q_{i}-\hat{q}_{i}|}{q_{i}}\ .$$ (25)

Last, we also use the kld to measure the divergence between the true and estimated label distributions.

Datasets

We use five node classification benchmark datasets and introduce different types of distribution shifts to evaluate the mentioned quantification methods. The datasets come from two different domains: Three citation network datasets, namely, CoraML, CiteSeer and PubMed [McCallum et al., 2000, Giles et al., 1998, Getoor, 2005, Sen et al., 2008, Namata et al., 2012], and two co-purchase datasets, namely Amazon Photos and Amazon Computers [McAuley et al., 2015, Shchur et al., 2019]. All reported results were obtained by averaging over 10 random splits of the node set into classifier-train/quantifier-train/test, with sizes of $5\%$/$15\%$/$80\%$ respectively. Since random splitting does not generate distribution shift, we synthetically introduce shifts in the test splits in two ways:

1. pps: To simulate pps, we randomly sample a set of Zipf distributions over the labels [Qi et al., 2020]. Given one such Zipf distribution, we sample vertices uniformly at random for each label so that the target label frequencies are reached.

2. Covariate Shift: This shift is introduced by uniformly sampling start vertices for each label and then performing a randomized bfs to sample a neighborhood of a fixed size around each start vertex.

In our experiments, both, the pps and the covariate shifted test splits each consist of 100 vertices. For each label, we sample 10 corresponding shifted splits and report the average results.

Classifiers

We use four different node classifiers to predict the labels of the vertices: A structure-unaware mlp, a gcn [Kipf and Welling, 2017], gat [Veličković et al., 2018], and appnp [Gasteiger et al., 2018]. All models are trained using the same training splits and hyperparameters, and two hidden layers/convolutions with widths of 64 and ReLU activations. Each model is trained ten times on each of the ten splits per dataset, totalling 100 models per dataset with which each quantifier is evaluated.

Quantifiers

sis and nacc are evaluated, both, separately and in combination. sis is evaluated using the sp kernel with $\lambda=\nicefrac{{1}}{{2}}$ and the ppr kernel from Eqs. 22 and 23.

4.2 Discussion of Results

CoraML
CiteSeer
Amazon Photos
Amazon Computers
PubMed
	(a) Hard Classifier	(b) Probabilistic Classifier

Figure 2: Quantification results for different dataset, classifier, quantifier combinations, compared to classifier accuracy.

Figure 2 compares the ae of the different quantification methods wrt. classifier accuracy on the shifted test data for three different distribution shift scenarios: no shift, pps and covariate shift.

Classifier Accuracy

The quality of the classifier has a significant impact on the quantification results, i.e., the error generally goes down with increasing classifier accuracy. Unsurprisingly, the structure-unaware mlp performs worst, while appnp generally yields the best results. Despite these differences, compared to the cc quantifier, the *acc-based methods are generally able to compensate the misclassifications of the weak classifiers, flattening the error curve.

Effectiveness of *sis

We observe that the difficulty of quantification depends on the type of distribution shift. In contrast, quantification without shift is trivial by definition, pps is generally easier than covariate shift, as it does not require sis and the choice of an appropriate vertex kernel. Under pps, sis with the ppr kernel generally performs worse than non-*sis methods since this kernel does not reflect the data-generating process. In contrast, under covariate shift, sis generally outperforms the non-*sis methods, demonstrating that kernel density estimation improves confusion estimates. However, on the CiteSeer dataset, sis performs significantly worse under covariate shift; this is likely because the ppr kernel is not well-suited for this dataset, highlighting the importance of choosing an appropriate kernel.

Effectiveness of *nacc

Since the goal of nacc is to improve class identifiability, via structural information, its effectiveness depends on the presence of non-homophilic regions which add collinearities to the confusion matrix. We find that nacc improves quantification results.

Table 1: Quantification using probabilistic classifiers (absolute error, relative absolute error and KL divergence).

[ column count=69, tabular=rl | rrr | rrr | rrr | rrr | rrr | rrr, separator=comma, table head=Model& CoraML CiteSeer Amazon Photos Amazon Computers PubMed Avg. Rank
& Shift Quantifier AE RAE KLD AE RAE KLD AE RAE KLD AE RAE KLD AE RAE KLD AE RAE KLD
, before reading=, table foot=, head to column names, late after line=
, ]tables/pcc.csv Section 4.2 shows probabilistic quantification results for the rae and kld metrics and sis with the sp kernel. The bold results indicate that there is no significant difference between the reported mean and the best mean within a given block, determined by the 95th percentile of a one-sided t-test. The experiments show that both sis and nacc are able to improve quantification results under pps and covariate shift.

5 Conclusion

We have introduced two novel graph quantification methods, sis and nacc; to our knowledge, this is the first work to investigate classifier-based graph quantification. sis enables quantification under covariate shift by estimating the test confusion matrix using a kernel density estimate of the test instance distribution. nacc uses the neighborhood structure of the graph to improve class identifiability. The effectiveness of our approach was demonstrated on multiple graph benchmark datasets.

We envision two lines of future research. First, in this work, we focused on extensions of acc to the graph setting. However, on tabular data, distribution matching quantifiers, such as DMy [González-Castro et al., 2013] or KDEy [Moreo et al., 2025], often outperform *acc-based approaches. An extension of distribution matching to gql could further improve the quantification performance on graphs. Second, as our experiments showed, choosing an appropriate kernel in sis is important. While the simple sp and ppr kernels generally perform well in our experiments, a deeper understanding of the practical applicability of vertex kernels for quantification is desirable. To this end, one could design an AutoML system, to automatically determine the type of distribution shift in a graph quantification problem in order to find an appropriate kernel based on that shift.