Advances in Set Function Learning: A Survey of Techniques and Applications

The first strategy for CNNs to learn set function is extending convolution operation to set. Wendler et al. [130] propose a novel class of CNNs for set functions by proposing powerset convolution and pooling. Powerset convolution is designed to be shift equivariant, meaning it commutes with specified shifts on the powerset domain. The shift is defined as modifying a set function \(s(A)\) by removing a subset Q from argument A, denoted as \(T_Qs=(s_{A\setminus Q})_{A\subseteq N}\), where N is the ground set. The convolution is given in Reference [93] as \((h\ast s)_A=\sum _{Q\subseteq N}h_Qs_{A\setminus Q}\), where the filter h is a set function. The powerset convolution layer is constructed by conducting powerset convolutions on multiple channels, summarizing the feature maps as in Reference [15], with both input and output being sets of set functions. Each output set function is derived by convolving the input set functions with corresponding filters, summing the results, and applying the non-linear transformation. Powerset pooling layer can reduce the complexity by aggregating elements into a smaller ground set, mapping the original set function to a new one on this reduced set, and can be implemented in various ways, such as combining elements as in Reference [109] and using a simple max rule. Powerset CNNs consist of multiple powerset convolution and pooling layers, the number of which can be adjusted according to specific tasks. However, the complexity analysis [74] shows that powerset CNNs are impractical for large ground sets as the number of set functions grows exponentially with the size of the ground set. Xu et al. [133] develop SpiderCNN, a novel CNN designed specifically for processing point clouds. The core component of SpiderCNN is the SpiderConv layer, which replaces conventional convolutional layers to enable convolution on point sets. Given a function F defined on a point set \(P\subseteq \mathbb {R}^n\) and a filter \(g:\mathbb {R}^n\rightarrow \mathbb {R}\) within a sphere centered at the origin with radius \(r\in \mathbb {R}\), the SpiderConv can be formulated aswhere \(p,q\) are points. Conventional convolution becomes a special case of SipderConv when \(P=\mathbb {Z}^2\) is a regular grid. In SpiderConv, the filter g belongs to a parameterized filter family \(\lbrace g_w\rbrace\), which is piecewise differentiable for w and can be efficiently optimized using stochastic gradient descent (SGD). \(\lbrace g_w\rbrace\) is defined as the product of a step function and a Taylor polynomial, enabling capturing local geodesic information and ensuring expressiveness. SpiderCNN inherits the advantages of CNNs, making it effective to extract deep features and achieve good performance in segmentation tasks.

The second strategy is combining CNN with symmetric aggregation like mean and max pooling, where the final aggregated feature depends only on the set contents and not their order. To deal with burst image deblurring, Aittla et al. [3] propose a U-Net-inspired [106] framework with symmetric pooling operations. In this framework, each image of the input set is processed individually through identical neural networks with tied weights, producing feature vectors. These feature vectors are computed through symmetry operations such as mean and max pooling. Eventually, the pooled features are processed through further neural network layers, outputting an estimate of the sharp image. In addition, the intermediate pooling layer is introduced, followed by \(1\times 1\) convolutions that fuse global features into local ones. This layer allows the concatenation of the pooled global state back to the local features, enabling information exchange between the set entities. Zhong et al. [151] design a CNN-based architecture called SetNet, which aggregates face descriptors into a compact descriptor. This framework is developed to enhance the efficiency and accuracy of retrieving a set of images that match a given query containing the descriptions of images for multiple identities. SetNet utilizes ResNet-50 [42] to extract features from each image, generating individual descriptors. These descriptors are aggregated into a fixed-length set-level vector using NetVLAD [4], which also helps reduce memory usage and runtime.

To handle learning tasks over set structured data, Zaheer et al. [140] construct a permutation-invariant model. For a countable set X and a set Y, the function \(f:X\rightarrow Y\) is a valid set function, i.e., invariant to the permutation of elements in X, if and only if it can be decomposed into the form: \(\rho (\sum _{x\in X}\phi (x))\), where \(\phi\) and \(\rho\) are appropriate transformations. As for an uncountable set X with fixed size M, any continuous function f defined on X, i.e., \(f:\mathbb {R}^{d\times M}\rightarrow Y\), is permutation-invariant if and only if f can be approximated arbitrarily closely by a function of the form \(\rho (\sum _{x\in X}\phi (x))\). Therefore, any set function f can be represented in this formulation:Based on this analysis, DeepSets is developed, capable of approximating any permutation-invariant function on the ground set X by using universal function approximators, such as neural networks, for the transformations \(\phi\) and \(\rho\). The model contains two main operations: (1) Each element \(x_m\) of the ground set X is transformed into a representation \(\phi (x_m)\) through the neural network \(\phi\). (2) The representations \(\phi (x_m)\) are summed to produce a single vector, which is then processed through network \(\rho\). The key idea is to aggregate all representations via summation and then apply nonlinear transformations through networks. In particular, the intermediate layers within DeepSets, such as \(\phi\), often exhibit permutation-equivariance, meaning that the processing of each element is independent of the order in which the elements are presented. This property ensures that the order of the elements does not affect their individual processing. Similarly to Definition 2.3, we formally define permutation-equivariance as follows.

The authors propose a novel formulation of permutation-equivariant functions, which can be represented as a neural network layer whose standard form is \(g_{\Theta }(\boldsymbol {x})=\sigma (\Theta \boldsymbol {x})\), where \(\Theta \in \mathbb {R}^{M\times M}\) is the weight vector and \(\sigma :\mathbb {R}\rightarrow \mathbb {R}\) is a nonlinear function such as sigmoid function. It is proved that \(g_{\Theta }:\mathbb {R}^M\rightarrow \mathbb {R}^M\) is permutation-equivariant if and only if all diagonal elements of \(\Theta\) are equal and all off-diagonal elements are tied together, i.e.,where \(\lambda ,\gamma \in \mathbb {R}\), \(\boldsymbol {1}=[1,\ldots ,1]^\top \in \mathbb {R}^M\), and \(\boldsymbol {I}\in \mathbb {R}^{M\times M}\) is a identity matrix. Therefore, the neural network \(g_{\Theta }(\boldsymbol {x})=\sigma (\Theta \boldsymbol {x})\) is permutation-equivariant if \(\Theta =\lambda \boldsymbol {I}+\gamma (\boldsymbol {11}^\top)\), i.e., \(g(\boldsymbol {x})=\sigma ((\lambda \boldsymbol {I}+\gamma (\boldsymbol {11}^\top)) \boldsymbol {x})\). The layer has several other variations when specifying the operations and parameters. In summary, the permutation-equivariant property of the intermediate layers in DeepSets ensures that each element is treated consistently regardless of its position in the set. The final symmetric aggregation combines these equivariant features in a permutation-invariant manner, ensuring that the order of elements does not affect the output.

There are some works generalizing DeepSets. Margon et al. [80] focus on a principled approach to learning from unordered set elements, particularly when the elements themselves exhibit inherent symmetries. The authors propose the Deep Sets for Symmetric elements (DSS) framework, which generalizes the DeepSets to accommodate additional symmetries of elements. The core innovation is the introduction of DSS layers, incorporating multiple linear layers L that are equivariant to the permutations of the set and the inherent symmetries of the set elements, such as translational symmetry in images and rotational symmetry in 3D shapes. This symmetry is represented by a group H that operates on the elements. Concretely, a DSS layer applies a transformation to each set element while also considering the aggregated information from the entire set. Based on Equation (3), the DSS layer for a set \(\lbrace x_1,\ldots ,x_n\rbrace \subseteq \mathbb {R}^d\) with symmetry group H and feature dimension d is defined bywhich generalizes DeepSets by applying linear H-equivariant functions \(L_1^H, L_2^H\). The authors prove that DSS networks are universal approximators, because the individual element-wise networks are universal for the symmetry group H, addressing the issue that restricting a network to be invariant or equivariant may reduce the expressive power [79]. Consequently, DSS layers can represent any function that respects the symmetries of the set elements and the set itself. In summary, the DSS framework extends DeepSets to problems involving symmetric elements, providing a comprehensive and theoretically grounded approach for learning from sets with intrinsic symmetries. Murphy et al. [83] propose Janossy pooling, a novel model for constructing permutation-invariant functions. Janossy pooling provides a universal method by representing a permutation-invariant function as the average of a permutation-sensitive function applied to all possible reorderings of the input sequence. However, the computational cost of summarizing all permutations and backpropagating gradients is pretty high. To solve this issue, the authors develop three approximation methods to trade off complexity and generalization: (1) canonical orderings: elements of the input sequence are reordered according to a predefined criterion, reducing the computational cost by avoiding the need to consider all permutations; (2) k-ary dependencies: the permutation-sensitive function is restricted to depend only on subsets of k elements at a time, reducing the number of permutations considered while still capturing important interactions; (3) permutation sampling: During training, permutations are randomly sampled, leading to fewer permutations. These strategies enable Janossy Pooling to unify and generalize existing methods, achieving competitive performance on various tasks compared to state-of-the-art techniques. Notably, DeepSets can be seen as a special case of Janossy Pooling with 1-ary dependencies, where the function depends on individual elements without considering interactions beyond simple aggregation. In contrast, Janossy Pooling allows for k-ary dependencies, capable of capturing higher-order interactions within the data.

There have been several works conducted to analyze the theoretical properties of DeepSets as it is a fundamental set function learning approach. Wagstaff et al. [122] refer to permutation-invariant function f represented by the formulation of Equation (2) as sum-decomposition, where the combination \((\rho ,\phi)\) of function \(\phi :\mathbb {R}\rightarrow Z\) and function \(\rho :Z\rightarrow \mathbb {R}\) is a sum-decomposition with latent space Z for function f, namely, the function f is sum-decomposable via Z. They analyze the limitations of enforcing permutation-invariance using sum-pooling and derive a necessary condition that a sum-decomposition-based model with universal function representation should satisfy. It is demonstrated that only if the dimension L of latent space where the summation is located is no less than the set size N, the sum-decomposition-based models can represent arbitrary continuous functions defined on a set with size N. To resolve the open question regarding the representation capabilities of high-dimensional DeepSets posed in Reference [122], Zweig et al. [153] conduct expressive power analysis of DeepSets. They indicate that DeepSets require an exponentially large width to approximate certain symmetric functions, implying that the dimension L of latent space grows exponentially with the size N and dimension D of the input set. This analysis demonstrates that DeepSets may be inherently inefficient for representing certain high-dimensional symmetric functions unless it is enhanced with mechanisms enabling interactions between set elements. Wang et al. [125] further reveal the relationship between the latent space dimension L and the expressive power of DeepSets [140], overcoming the limitations of previous works that focus solely on one-dimensional features or complex analytic activations, which are impractical due to the exponential growth of L with N and D. Considering high dimensional features, i.e., \(D\gt 1\), the bounds of minimal latent space dimension L are proved to be divided into two categories according to the encoding network \(\phi\): (1) If \(\phi\) applies a linear layer with power mapping, then we can get \(N(D+1)\le L\lt N^5D^2\). (2) If \(\phi\) applies a linear layer and an exponential activation function, then we can get a tighter bound, \(ND\le L\le N^4D^2\). The proposed bounds imply that it is sufficient to model the latent space of DeepSets with L being \(\mathrm{poly}(N,D)\) for the universal approximation of set functions. It is also demonstrated that continuous mappings \(\phi\) and \(\rho\) are crucial for ensuring universal approximation of DeepSets. Table 2 compares the lower bounds of different research.

There are some works trying to propose novel aggregating methods for set function learning. Aggregating inputs into a single representation is a common mechanism in set function learning, such as DeepSets, which utilizes sum-pooling to aggregate element-wise embeddings. Inspired by DeepSets, Abedin et al. [1] employ a set-based deep learning approach called SparseSense to handle the sparse data from passive sensors in human activity recognition (HAR) tasks. Unlike traditional methods that require dense data streams or rely on interpolation to estimate missing data points, SparseSense processes the sparse data directly, mitigating large estimation errors and long recognition delays. This method regards sparse sensor data as sets, allowing the model to focus on extracting discriminative features of all activity categories without relying on temporal correlations. The key idea is to apply a shared embedding network to project each set element into a higher-dimensional space, followed by featurewise maximum pooling to aggregate these embeddings into a fixed-size global representation for activity classification. SparseSense extends DeepSets to HAR, demonstrating that set-based neural networks can effectively handle irregular data points and tolerate missing information. Bartunov et al. [12] introduce an optimization-based aggregation method named Equilibrium Aggregation. This method generalizes existing pooling-based approaches, overcoming the limitations of existing techniques such as sum-pooling, which are constrained by their representational power. The Equilibrium Aggregation models the potential function \(F_\theta (x,y)\), which quantifies the discrepancy between each set element x and aggregation result y, as a learnable neural network with parameter \(\theta\). This layer architecture can be integrated into another multi-layer neural network to aggregate sets. The energy-minimization of Equilibrium Aggregation can be formulated aswhere \(R_\theta (y)\) is regularization. The aggregation result y is computed through solving Equation (4) with numerical methods such as gradient descent. The neural network framework with Equilibrium Aggregation can be formulated as \(\rho (\phi _\theta (X))\) where \(\rho\) is a neural network. It is theoretically proved that this framework can universally approximate any continuous permutation-invariant functions if the output of Equation (4) has the same size as the input set. Equilibrium Aggregation provides a more flexible framework than DeepSets by utilizing a learnable potential function, potentially achieving better performance in tasks that require more detailed data representation. Horn et al. [43] propose a novel framework called Set Functions for Time Series (SeFT) for classifying irregularly sampled time series. SeFT regards time-series data as a set of observations, addressing the issues of irregular sampling and unaligned measurements without requiring imputation. This model employs a set function \(f:S\rightarrow {\mathbb {R}^c}\) derived from Equation (2). Denoting \(s_j\) as a single observation of the time series S, the function f can be formulated aswhere \(h:\Omega \rightarrow \mathbb {R}^d\) and \(g:\mathbb {R}^d\rightarrow \mathbb {R}^c\) are both neural networks, with h mapping observations from the domain \(\Omega\) to a d-dimensional latent space, and g further mapping this latent representation to the final c-dimensional classification space. The variant of positional encoding [120] is used for time encoding, employing multiple trigonometric functions at different frequencies to convert 1-dimensional time t of each observation into a multi-dimensional input. To handle large observation sets and highlight the most relevant data points, a weighted mean aggregating approach based on scaled dot-product attention with multiple heads is designed to weigh different observations. This aggregating method independently calculates each element’s embedding, achieving a runtime and memory complexity of \({O}(n)\). The SeFT extends the representation of DeepSets specifically to the time series with irregular sampling, where the order of observations is not fixed and might not follow a regular interval.

There are multiple works proposing methods that build on the foundational concepts of DeepSets, extending these concepts to specific scenarios. Yi et al. [137] introduce CytoSet designed to deal with sets of cells and predict the clinical outcome of patients. Since the order of cells’ profile has no biological relevance in flow and mass cytometry experiments, CytoSet regards the cytometry data as a set and extracts information through a permutation-invariant neural network based on DeepSets. This approach predicts the clinical outcome from the patient sample represented as a set of cells, with each cell characterized by a vector of protein measurements. In the proposed model, several permutation-equivariant blocks, as described in Reference [140], are stacked to transform the representation of each set element. The output of these blocks is processed by max-pooling, which measures the presence of high response cells and produces an embedding vector for the set. This vector is then passed through fully connected layers to predict the clinical outcome. CytoSet extends the concept of DeepSets to clinical cytometry data analysis, generalizing CellCNN [7] and CytoDx [46], and achieving better experiments performance compared to them. Ou et al. [86] present equivariant variational inference for set function learning (EquiVSet) to predict set-valued outputs (subsets) that optimize a certain utility function over a given ground set under the optimal subset (OS) supervision oracle, where the optimal subset provides the maximum utility. They combine energy-based method with DeepSets to construct an appropriate set mass function that increases monotonically with a set utility function. To enable training models on varying ground sets and overcome the instability caused by the high dimension of sets when directly optimizing likelihood, a scalable training and inference algorithm is proposed by utilizing the maximum likelihood principle in conjunction with mean-field inference as a surrogate. The EquiVSet improves DeepSets to model the utility function explicitly and handle more complex tasks involving OS oracles. Wang et al. [124] develop an effective model termed as DTS-ERA, which combines the proposed Deep Temporal Sets (DTS) with Evidential Reinforced Attentions (ERA) to uncover the signature behavioral patterns of multimodal data in behavior analysis of children with autism spectrum disorder. DTS-ERA is implemented in the manner of few-shot learning, enabling it to effectively handle situations with limited data. DTS is a multimodal version of DeepSets, capable of capturing complex temporal and spatial relationships in multimodal data. It is composed of a temporal encoder and a spatial encoder, which generate feature representations that maintain temporal dependencies and spatial locality. These feature representations are then concatenated and aggregated through an average-pooling to obtain the deep-set encoding. In ERA, DTS is combined with reinforcement learning agent where an evidential reward function is designed to learn an epistemic policy, which selects representative embeddings as attention signatures. ERA incorporates evidential learning to estimate uncertainty, allowing the model to distinguish between known and unknown regions effectively, thereby improving the reliability of the predictions.

PointNet can directly process point clouds without converting them into regular data structures such as 3D voxel grids, maintaining the inherent properties of point clouds. The components of PointNet are similar to DeepSets while only replacing the sum-pooling with max-pooling. For a finite point set X and its element x, the set function \(f:2^X\rightarrow Y\) whose value corresponds to the semantic label of point set can be approximated by the PointNet formulated aswhere \(\phi\) captures features of each point in X. These features are aggregated through max-pooling and then passed to \(\rho\) to obtain the output. Both \(\rho\) and \(\phi\) are neural networks or other parameterized models with learnable parameters. This framework is permutation-invariant, because max-pooling can process arbitrary orders of points in the point set and obtain the same output. It is theoretically proved that PointNet is capable of approximating any continuous set function if the max-pooling layer contains enough neurons.

Chrisian et al. [18] explore the expressive power of neural networks that use set pooling mechanisms. The authors introduce and analyze a variety of set pooling architectures, such as sum-pooling (DeepSets), max-pooling (PointNet), and average-pooling (normalized-DeepSets). The theoretical analysis reveals that PointNet cannot generally approximate averages of continuous functions over sets (e.g., center-of-mass), and DeepSets is strictly more expressive than PointNet in the constant cardinality setting. This finding implies that the choice of set pooling function has a dramatic impact on the expressiveness of these networks. Unexpectedly, it is also proved that any function that can be uniformly approximated by both PointNet and normalized-DeepSets should be constant under the unbounded cardinality setting.

There are several works that enhance PointNet, extending its applicability to more complex scenarios. To overcome the limitation that PointNet cannot learn the local structures at various scales, Qi et al. [96] develop a hierarchical neural network named PointNet++, which applies PointNet recursively to nested partitions of the input point set. PointNet++ contains multiple set abstraction levels, including sampling layer, grouping layer, and PointNet layer. The sampling layer chooses points to define local regions’ centroids, around which the grouping layer explores neighboring points to build local regions. Then the PointNet layer encodes local region patterns into feature vectors. In particular, the grouping layer has two implementations: multi-scale grouping and multi-resolution grouping. These methods are capable of adaptively aggregating multi-scale features with respect to corresponding point densities, thereby eliminating the impact of varying point set densities on different regions. Generally, PointNet++ begins by extracting local features that capture fine geometric structures within small neighborhoods through PointNet. These local features are then grouped into larger units and further processed to generate higher-level features. Such hierarchical process is repeated iteratively until the comprehensive features of the entire point set are obtained, realizing both robustness and detail capture. While PointNet uses a global max-pooling operation to aggregate features from the entire point set, PointNet++ enhances it by introducing a multi-scale hierarchical learning process, expanding its capabilities to capture detailed local structures and handle varying point densities. In contrast to PointNet that directly processes point cloud by considering each point independently and utilizing max-pooling to aggregate global features, Prokudin et al. [92] design a type of residual representation termed as basis point sets (BPS), which can encode a point cloud into a fixed-length vector, enabling the use of standard machine learning techniques. To construct BPS, the point clouds are normalized to fit a unit ball with radius \(r\in \mathbb {R}\), where we randomly sample \(k\in \mathbb {R}\) points from a uniform distribution to obtain basis point set. By calculating the minimal distance from each basis point to the nearest point in the point cloud, we obtain feature vectors for every point cloud. These feature vectors can be taken as inputs of the learning algorithms. The point cloud classification experiments demonstrate that the framework combining MLP with BPS achieves performance comparable to PointNet, while significantly reducing the number of parameters and computational complexity.

Set Transformer [64] is an attention-based neural network method capable of modeling interactions across input set elements, which are often overlooked by set pooling methods such as DeepSets [140] and PointNet [95]. Based on Transformer [120], Set Transformer employs permutation-invariant self-attention to capture pairwise and higher-order interactions between elements. Suppose that \(Q,R\in \mathbb {R}^{n\times d}\) are query set and value set respectively, consisting of n d-dimensional vectors. To construct the Set Attention Block (SAB), the authors employ the Multihead Attention Block (MAB), which is a variant of the Transformer’s encoder, with positional encoding and dropout removed. Given matrices \(X,Y\in \mathbb {R}^{n\times d}\), the MAB with parameter \(\omega\) is defined as follows:where \(\mathrm{LN}\) is layer normalization [8] and \(\mathrm{rF}\) is an arbitrary row-wise feed-forward layer. The SAB can be formulated as \(\mathrm{SAB}(X)=\mathrm{MAB}(X,X)\). The higher-order interactions of elements can be captured by stacking multiple SABs. To reduce the high computational cost associated with self-attention, the Induced Set Attention Block (ISAB) is designed based on SAB and inspired by inducing point methods used in sparse Gaussian processes. The ISAB containing m inducing points I, i.e., m trainable d-dimensional vectors \(I\in \mathbb {R}^{m\times d}\), can be formulated aswhere h is permutation-equivariant to X and \(\mathrm{ISAB}_m(X)\) is permutation-invariant to X. In this way, the computational time is reduced from \(O(n^2)\) in SAB to \(O(mn)\) in ISAB. The Pooing by Multihead Attention (PMA) with k seed vectors \(S\in \mathbb {R}^{k\times d}\), i.e., \(\mathrm{PMA}_k(Z)=\mathrm{MAB}(S,\mathrm{rF}(Z))\), is developed to aggregate encoding features set \(Z\in \mathbb {R}^{n\times d}\). This mechanism allows the model to adaptively weigh the importance of different elements in the set, which is particularly useful in scenarios requiring multiple correlated outputs, such as clustering tasks. Generally speaking, in Set Transformer, the input set is encoded by the stacking of SABs or ISABs, followed by aggregation using PMA. The aggregated representation is then passed through a feed-forward network to produce the output. It is theoretically proved that Set Transformer is capable of universally approximating any set function.

Through gradient analysis, Zhang et al. [144] indicate that DeepSets and Set Transformer probably suffer from vanishing and exploding gradients when stacking more layers. They also observe that layer normalization discourages performance, because the invariance of layer norm decreases the representation power and drops potentially useful information for prediction. To tackle such issues and make these set neural networks deeper, DeepSets++ (DS++) and Set Transformer++ (ST++) are developed by proposing equivariant residual connections (ERC) and set norm. ERC is a refined residual connection approach adhering to the clean path principle, capable of avoiding potential gradient issues by maintaining a clean path from input to output. The set norm is a novel normalization layer, which standardizes each set over the minimal number of dimensions and transforms features individually. This mechanism preserves most of the mean and variance information, avoiding the invariance issues associated with layer normalization. By integrating ERC and set norm into the encoders of DeepSets and Set Transformer, respectively, the enhanced models DS++ and ST++ are constructed. These improvements enable the models to achieve greater depth and comparable performance, effectively addressing the instability issues in the original versions.

There are several works employing the Set Transformer as encoders to construct new models for set function learning. Li et al. [67] propose an effective recipe representation learning model named Reciptor, which jointly processes ingredients and cooking instructions. The ingredient set is encoded by Set Transformer, enhancing the model’s ability to capture interdependence among elements. A pretrained skip-instruction model is employed to encode the cooking instructions, generating initial embeddings and providing a context-aware representation of the entire cooking process. These initial embeddings are subsequently processed by a forward long short-term memory network to produce the final instruction embeddings. To further optimize the learned embeddings, the authors utilize a novel knowledge graph-based triplet sampling loss [22], ensuring that semantically related recipes are closer in the latent space. The embeddings are refined by combining a triplet loss with a cosine similarity loss between ingredient and instruction embeddings. The Reciptor outperforms baselines on two newly designed downstream classification tasks. Based on Set Transformer, Gim et al. [38] design a set-based cooking recommender called RecipeBowl, which processes a given set of ingredients and cooking tags to output corresponding ingredient and recipe choices. Set Transformer is employed as the encoder to build a comprehensive representation of the ingredient set. A two-way decoder maps the representation into two distinct embedding spaces: one for predicting missing ingredients and the other for recommending relevant recipes. The model is trained using a combination of negative likelihood loss based on Euclidean distances and cosine embedding loss for the recipe prediction task, ensuring that the predicted ingredients and recipes are aligned with their actual counterparts in the embedding space.

Zhang et al. [142] present efficient algorithms to address the challenges of relational reasoning in cooperative Multi-Agent Reinforcement Learning (MARL) with permutation-invariant agents. They leverage Set Transformer to implement complex relational reasoning among agents in MARL. Two algorithms are proposed, including model-free and model-based offline MARL algorithms. The model-free approach employs transformers to estimate the action-value function, incorporating a pessimistic policy to handle distributional shifts in offline settings. The model-based approach estimates the system dynamics with transformers, also utilizing a pessimistic policy. The key contribution is deriving generalization error bounds for transformers in MARL, demonstrating that these bounds are independent of the number of agents and less sensitive to the depth of the network. Jurewicz et al. [55] develop Set Interdependence Transformer (SIT), an efficient set encoder to solve set-to-sequence tasks. This set-to-sequence model is established by combining the SIT with a permutation decoder. Set transformer serves as the basic set encoder, learning permutation-equivariant representations of individual elements and permutation-invariant representations of the entire set. SIT enhances these representations with an augmented attention mechanism to capture higher-order interdependencies. The permutation decoder uses an improved pointer attention mechanism to select elements, forming coherent output sequences. This approach effectively handles sets of varying cardinalities and generalizes well to unseen set sizes, as shown in experiments.

Lee et al. [66] propose Meta-Interpolation, a universal task augmentation method designed for few-task meta-learning. Meta-Interpolation utilizes Set Transformer to process the embeddings of support and query sets from different tasks and learn a parameterized set function, mapping sets of task embeddings to new embeddings that mix features from different tasks. This process creates new tasks that have unique features drawn from the tasks being interpolated. Bilevel optimization is employed to jointly optimize parameters of the meta-learner and the set function. The upper-level optimization aims to minimize the loss on meta-validation tasks, ensuring that this augmentation strategy improves generalization. The lower-level optimization adapts the meta-learner to augmented tasks, reducing the risk of overfitting to the limited meta-training set. This method theoretically regularizes the meta-learner by enforcing a distribution-dependent regularization, which decreases the Rademacher complexity and thus improves the generalization.

There are multiple works utilizing attention mechanisms to learn set functions, similar to Set Transformer. Girgis et al. [39] develop an encoder–decoder framework, Latent Variable Sequential Set Transformers, termed as AutoBots, to deal with the challenging task of predicting the future trajectories of multiple interacting agents. The permutation-equivariant encoder processes sequences of sets representing the agents states over time, incorporating both temporal and social information through multiple Multi-Head Self-Attention modules. The decoder utilizes multiple matrices of learnable seed parameters, enabling the model to capture multi-modal nature of future trajectories. This process allows for the generation of diverse and socially consistent predictions across the entire scene in a single forward pass. The model achieves state-of-the-art performance, particularly in trajectory predictions that adhere to real-world constraints such as road layouts. Zhao et al. [150] propose Point Transformer, a novel architecture tailored for unordered 3D point sets. Point Transformer mainly contains SortNet and local-global attention module. SortNet is a novel neural network that learns to sort the input point cloud data into a specific order based on selected features. Once the points are sorted, the Point Transformer layer applies local attention to aggregate features of each point from its k nearest neighbors, capturing fine-grained details and local geometric structures within small regions. Following the local attention, global attention aggregates features from the entire point cloud or larger regions, complementing the local information and enabling the network to understand the overall structure. The outputs from the local and global attention modules can be combined, either through concatenation or a weighted sum, to form a comprehensive feature representation for each point. This representation can be used in downstream tasks for learning the underlying shape.

DSPN [145] is a model designed to predict sets from feature vectors, addressing the issue that previous methods such as RNNs result in discontinuous and inaccurate predictions due to lack of consideration on the unordered nature of sets. DSPN employs the same module for both encoding and decoding processes. Concretely, the encoder \(g_{\text{enc}}\) maps the input set X into the latent space z, i.e., \(z=g_{\text{enc}}(X)\), obtaining the representation of X. The decoder \(g_{dec}\) predicts a set from this representation, i.e., \(\hat{X}=g_{\text{dec}}(z)\), applying gradient descent with a learnable initial guess to find a set whose latent representation matches the input set. This process can be regarded as a nested optimization. In the inner loop, the predicted set is refined iteratively to minimize the difference between its encoding and the target representation. Meanwhile, in the outer loop, the weights of the model are trained by minimizing the loss between the predicted set and the true set. Formally, the representation loss and decoder are defined aswhere the permutation-invariant \(L_{\text{repr}}\) compares \(\hat{X}\) with the latent representation of X. Considering \(g_{\text{enc}}\) is a neural network, the gradient descent is utilized for T steps to solve the minimization of Equation (7) from initial set \(\hat{X}^{(0)}\). At the same time, the weights of \(g_{\text{enc}}\) are trained to minimize set loss \(L_{\text{set}}(\hat{X}^{(T)},Y)\), where the \(L_{\text{set}}\) can represent Chamfer loss or pairwise loss, to obtain an appropriate representation z. In general set prediction, there is no set encoder, since the input is usually a vector instead of a set, in which case, a term is added to the loss of outer loop to ensure \(g_{\text{enc}}(X)\approx z\). The DSPN shows significant improvements over traditional methods, particularly in providing accurate set predictions without requiring complex postprocessing. This work opens up new possibilities for set prediction problems.

There are several works that improve DSPN and extend its application to specific scenarios. Zhang et al. [146] design a differentiable set pooling method called FSPool, which consists of two operations, sorting and weighted summation. Instead of treating set elements as whole units, FSPool sorts each feature independently across the set elements. After sorting, a weighted sum is computed, with the weights determined by a learnable calibrator function. FSPool can handle sets of different sizes using a continuous representation of weights. In experiments involving both bounding box and state prediction, the authors combine DSPN and other models, such as MLP, with FSPool, max-pooling, and sum-pooling respectively. The results indicate that simply replacing the pooling function in an existing model with FSPool leads to better results and faster convergence. By replacing the gradient descent updates of DSPN with transformer that provides more expressive and efficient updates, Kosiorek et al. [58] propose Transformer Set Prediction Network (TSPN), where an MLP is utilized to predict the number of points from the input embedding and decide the size of the initial predicted set. TSPN initializes the predicted set with a random set of points sampled from a learned distribution, enhancing flexibility. The transformer is employed to iteratively update set elements, leveraging self-attention mechanism to model dependencies between elements and output the predicted set. Compared to DSPN, TSPN achieves more expressiveness and less computational cost. Zhang et al. [141] develop a framework called Deep Energy-based Set Prediction (DESP), which treats set prediction as a problem of conditional density estimation rather than optimization with set-specific losses. This method utilizes deep energy-based models to capture the distribution of sets given some input features. The energy function \(E_\theta (x,Y)\) assigns a scalar energy to a pair of input features x and a set Y, where a lower energy indicates a higher likelihood of the set. Given the input, the probability of a set is \(P_\theta (Y|x)=\frac{1}{Z(x;\theta)}\exp (-E_\theta (x,Y))\), where \(Z(x;\theta)\) is a partition function. In the proposed framework, two permutation-invariant energy functions \(E_{DS}(x,Y)\) and \(E_{SE}(x,Y)\) are derived from DeepSets and DSPN, respectively. These energy functions can be used to formulate deep energy-based models, which can be trained by minimizing the negative log-likelihood, allowing for the approximation of the true data distribution without requiring explicit pairwise comparison between predicted and ground truth sets. DESP utilizes a stochastically augmented prediction algorithm, which helps explore multiple modes to generate diverse outputs. The final predicted set is determined as the set with the lowest energy among all captured sets. DESP extends the capabilities of DSPN to more effectively handle the inherent complexity and stochasticity of real-world tasks.

Zhang et al. [148] define a relaxation of common set equivariance, the multiset equivariance, which does not require equal elements in a multiset to remain equal after transformation. This property is crucial for handling multisets with duplicate elements, enabling models to process them with different strategies. Additionally, the authors propose exclusive multiset equivariance, which describes models that are multiset equivariant but not set equivariant, aiming to tackle the issue that set-equivariant functions cannot represent certain functions on multisets. It is proved that DSPN satisfies the exclusive multiset equivariance when selecting the appropriate set encoder. To reduce memory and computational requirements, the implicit DSPN (iDSPN) is developed by employing approximate implicit differentiation to replace the gradient descent of DSPN. This method avoids storing intermediate gradient steps by directly computing the gradient at the optimal point, making the optimization process more efficient. iDSPN shows superior performance compared to traditional set-equivariant models, especially in handling multisets and large-scale set prediction tasks.

DSFs [28] are a special class of submodular functions that strictly generalizes many existing submodular functions and inherits some properties from them. For example, DSF can represent decomposable submodular functions, which can be expressed as sums of concave functions composed with modular functions. A notable subclass of these functions is the feature-based submodular function [128], which can be formulated as \(f(X)=\sum _{u\in U}w_u\phi _u(m_u(X))\), where \(\phi _u\) denotes non-deceasing univariate normalized concave function, \(m_u\) represents feature-specific modular function and \(w_u\) is feature weight, all of which are non-negative. To overcome the limitation that features themselves cannot interact in feature-based submodular functions, an additional layer of nested concave functions is employed, i.e., \(f(X)=\sum _{s\in S}\omega _s\phi _s(\sum _{s\in U}w_{s,u}\phi _u(m_u(X)))\), where S denotes a set of meta-features, \(\omega _s\) represents meta-feature weight and \(\phi _s\) is a non-decreasing concave function. The term \(w_{s,u}\) represents the feature weight corresponding to meta-feature s. By recursively applying the above layer, we can derive the DSF. Consider a series of disjoint sets \(V^{(0)},V^{(1)},V^{(2)},\ldots ,V^{(K)}\), where \(V^{(0)}\) is the ground set, \(V^{(1)}\) is features set, \(V^{(2)}\) is meta-features set, \(V^{(3)}\) is meta-meta-features set and by analogy until \(V^{(K)}\), with each set representing a layer. Denoting the size of \(V^{(i)}\) as \(d^i=|V^{(i)}|\), we can employ a matrix \(w^{(i)}\in \mathbb {R}_+^{d^i\times d^{i-1}}, i\in \lbrace 1,2,\ldots ,K\rbrace\) to connect two continuous layers. The element at row \(v^i\) and column \(v^{i-1}\) of matrix \(w^{(i)}\) is denoted by \(w_{v^i}^{i}\), where \(w_{v^i}^{i}:V^{i-1}\rightarrow \mathbb {R}_+\) represents a modular function over set \(V^{i-1}\). In this case, the matrix includes \(d^i\) such modular functions. Moreover, given a non-negative non-decreasing concave function \(\phi _{v^k}:\mathbb {R}_+\rightarrow \mathbb {R}_+\) and any set \(A\subseteq V\), a K-layer DSF \(f:2^V\rightarrow \mathbb {R}_+\) can be formulated asHaving shown the definition of DSF, it can be seen that DSF is composed of multiple layers, with each layer taking a positive linear combination of the previous layer’s outputs followed by the application of a concave function. This hierarchical structure enables DSF to capture complex interactions within the data. The layered structured DSF shares similarities with deep neural networks (DNN), allowing for extending DNN learning techniques to DSF. The authors utilize a max-margin learning approach that is tailored to maintain the submodularity for training DSFs. This learning process involves adjusting the parameters of the DSF to maximize a margin-based objective function, ensuring that the learned function assigns high values to desired subsets while penalizing undesired ones.

There are some works that extend DSF to specific scenarios and improve its capabilities. Ghadimi et al. [35] propose a novel model called deep submodular network (DSN), which combines the principles of deep learning with submodular optimization for multi-document summarization. DSN is similar to DSF, with the key difference being that DSN employs both modular and submodular functions to construct network blocks, whereas DSF only utilizes modular functions. Consequently, DSN generalizes DSF, making it applicable in a wider range of scenarios. The DSN utilizes the L-BFGS-B algorithm [20] for training, which is memory-efficient and suitable for maintaining non-negative weights. Manupriya et al. [76] design a novel approach called Submodular Ensembled Attribution for Neural Networks (SEA-NN), which aims to interpret the contribution of each input feature to the neural network’s output, particularly in image-based tasks. The core component of SEA-NN is a submodular score function learned by DSF, which achieves the combination of several existing gradient-based attribution methods, such as Integrated Gradients and Smooth Integrated Gradients, and integrates their score bias. It is trained using heatmaps generated by baseline attribution methods, to increase the score for features that are highly relevant and specific. The learned scoring function re-evaluates the importance of features in the input by assessing the marginal gain of each feature, reducing the attribution scores of redundant features that may be present in the raw attribution maps. The SEA-NN is model-agnostic and can be applied to various scenarios.

DSF models submodular functions as an aggregation of modular concave functions, but it does not provide methods for selecting these concave methods, complicating the practical application. To eliminate this gap, De et al. [25] introduce novel neural networks, FLEXSUBNET, to estimate both monotone and non-monotone submodular functions. FLEXSUBNET models submodular functions by recursively applying concave functions to modular functions. This neural network allows for learning these concave functions from data, enhancing the expressiveness. The core of FLEXSUBNET is a simple recursive chain that is a restriction of complex topology, with each node in the chain sharing the learnable concave function. According to the monotonicity of learned functions, two scenarios are considered: (1) For monotone submodular function, it is learned through a recursive model. At each step, the model calculates a linear combination of a previously computed submodular function and a modular function, which is then processed by a learnable concave function to produce a composed submodular function. (2) For non-monotone submodular functions, it is also learned through a recursive model, where a non-monotone concave function is applied to a modular function. The model can be trained using (set, value) pairs or (perimeter-set, high-value-subset) pairs, with applications in subset selection tasks where high-value subsets need to be extracted from larger sets.

There are some datasets commonly used for point cloud processing tasks. The ModelNet40 dataset [132] consists of 12,311 CAD models from 40 categories of man-made objects. The ShapeNet dataset [138] contains 16,881 3D shapes from 16 categories, each annotated with 50 distinct parts. The Stanford 3D semantic parsing dataset [5] includes Matterport 3D scans of 271 rooms across six areas, annotated with 13 semantic labels like chair, table, and floor. The Point Cloud MNIST 2D dataset converts MNIST [63] images into 2D point clouds, comprising 60,000 training and 10,000 testing samples, with each set containing 34–35 points. The Oxford Buildings Dataset [90] contains 5,062 images of 11 Oxford landmarks, each with 55 queries for evaluating object retrieval systems.

We summarize some image datasets for anomaly detection, set retrieval, and multi-label classification. CelebA dataset [73] contains 202,599 celebrity face images annotated with 40 Boolean attributes, such as “smiling,” “wearing glasses,” and “blonde hair.” The Celebrity Together dataset [151] includes 194,000 images with 546,000 labeled faces, averaging 2.8 faces per image. The MS COCO dataset [70] comprises 123,000 images labeled with per-instance segmentation masks of 80 classes. Each image includes 0 to 18 objects, with most containing 1 to 3 labels.

The following datasets are used to evaluate set function learning methods in recommendation systems. Amazon baby registry dataset [36] contains 29,632 baby registries, each listing 5 to 100 products categorized into groups like “toys” and “furniture.” The Recipe1M dataset [77] consists of 1,029,720 cooking recipes with ingredients, instructions, images, and 1,047 semantic categories parsed from titles, covering 507,834 recipes.

Here are some datasets used for molecular property and hematocrit level prediction. The Flow-RBC dataset [144] contains 98,240 training and 23,104 test sets, each representing 1,000 red blood cells with volume and hemoglobin content measurements. PDBBind dataset [72] provides experimental binding data for 10,776 biomolecular complexes, including 8,302 protein–ligand and 2,474 other complexes. The BindingDB9 dataset is a public database of measured binding affinities, consisting of 52,273 drug-targets with drug-like small molecules.

The following datasets can be used for object detection and set property prediction. SHIFT15M [56] contains 15 million images and videos captured in diverse driving environments with annotations for object bounding boxes, instance segmentation masks, and semantic labels, covering vehicles, pedestrians, road signs, and more. CLEVR [52] is a visual question answering dataset with 70,000 training images and 700,000 questions, plus additional validation and test sets. Questions fall into five types: existence, counting, integer comparison, attribute queries, and attribute comparisons. Each scene contains 3D-rendered objects characterized by size, shape, material and color, forming 96 unique combinations.