Restricted Indian buffet processes

Abstract

Latent feature models are a powerful tool for modeling data with globally-shared features. Nonparametric distributions over exchangeable sets of features, such as the Indian Buffet Process, offer modeling flexibility by letting the number of latent features be unbounded. However, current models impose implicit distributions over the number of latent features per data point, and these implicit distributions may not match our knowledge about the data. In this work, we demonstrate how the restricted Indian buffet process circumvents this restriction, allowing arbitrary distributions over the number of features in an observation. We discuss several alternative constructions of the model and apply the insights to develop Markov Chain Monte Carlo and variational methods for simulation and posterior inference.

Appendix: Impact of truncation level on approximation quality when simulating from the R-IBP

In Sect. 5, we described two approximate methods for sampling from the R-IBP, that made use of a finite-dimensional approximation to the beta process-distributed measure $\mu$. As the dimensionality I of the approximation tends to infinity, these methods will give exact samples from the R-IBP; however a fixed finite I will introduce errors. In this appendix, we discuss the errors introduced in both cases, and provide an error bound for the inclusion probability sampler.

1.1 Impact of truncation level in an inclusion probability sampler

If a size-ordered stick-breaking representation is used to approximate the weights $\pi$, then we can directly bound the errors on the inclusion probabilities as functions of the truncation level I, the size of the smallest instantiated weight $\pi _I$, and the function f. To do so, we first expand the expression for the probabilities $S^\infty _J$, starting with Eq. 15:

$$\begin{aligned} \textstyle S_J^\infty= & {} \sum _{s \in A_J(I)} \prod _{k \in s} \pi _k \prod _{j \ni s} ( 1 - \pi _j )\\&+ \sum _{s \ni A_J(I)} \prod _{k \in s} \pi _k \prod _{j \ni s} ( 1 - \pi _j ) \\= & {} \exp (-\pi _I\alpha ) \sum _{s \in A_J(I)} \prod _{k \in s} \pi _k \prod _{j \ni s, j \le I} ( 1 - \pi _j )\\&+ \sum _{s \ni A_J(I)} \prod _{k \in s} \pi _k \prod _{j \ni s} ( 1 - \pi _j ) \\= & {} \exp (-\pi _I\alpha ) S^I_J + \sum _{s \ni A_J(I)} \prod _{k \in s} \pi _k \prod _{j \ni s} ( 1 - \pi _j ) \end{aligned}$$

where $A_J(I)$ are the sets of feature allocations in which all J instantiated features are associated with one of the I largest atoms in $\mu$. The second line follows because the probability of that none of the features associated with the remaining atoms are selected is $\exp (-\pi _I \alpha )$.

Since the probability that at least one feature outside the most significant I features appears is $1 - \exp (-\pi _I\alpha )$, the second term is bounded between 0 and $1 - \exp (-\pi _I\alpha )$. Thus we can bound the inclusion probabilities

$$\begin{aligned}&\eta _{k;J}= \pi _k \frac{ S_{J-1}^{\infty }( \pi _1,...,\pi _{k-1},\pi _{k+1},...,\pi _I ) }{ S_J^{\infty }( \pi _1,...,\pi _I ) } \\&\quad \ge \pi _k \frac{ e^{-\pi _I\alpha }S_{J-1}^{I-1}( \pi _1,...,\pi _{k-1},\pi _{k+1},...,\pi _I ) }{ e^{-\pi _I\alpha }S_J^{I}( \pi _1,...,\pi _I ) + ( 1 - e^{-\pi _I\alpha } ) }\\&\quad \le \pi _k \frac{ e^{-\pi _I\alpha }S_{J-1}^{I-1}( \pi _1,...,\pi _{k-1},\pi _{k+1},...,\pi _I ) + ( 1 - e^{-\pi _I\alpha }) }{ e^{-\pi _I\alpha }S_J^{I}( \pi _1,...,\pi _I ) } \end{aligned}$$

As expected, the quality of the approximation depends not only truncation I (and associated $\pi _I$) but also on the values $S^I_J$. If the probability of sampling J elements from the first I is low, then the approximation will be poor because it is likely that additional features would have been required to sample J elements. These bounds can be used in situations where one can use approximate, rather than exact, probabilities.

1.2 Impact of truncation level in a rejection sampler

As $I\rightarrow \infty$, both the weak-limit approximation of Eq. 12 and the stick-breaking construction of Eq. 13 will give exact samples from the R-IBP. However, a finite I will introduce errors. When a stick-breaking representation for $\mu$ is used, then we know that all weights $\pi _j$, $j > I$ will be less than $\pi _I$. In particular, the iterative nature of the stick-breaking construction means that, if we exclude the first I atoms $\pi _1,\dots , \pi _I$, and scale the remaining atoms by $\pi _I$, we are left with a (strictly ordered) sample from the beta process.

We can consider the error introduced by this construction by considering the values of $z_{nj}$ that are excluded due to the truncation. If there are any non-zero elements $z_{nj}$ for $j > I$, our rejection probability will not be correct. Since the weights $\pi _j, j>I$ are described by a scaled beta process, we know that the number of excluded non-zero elements will be distributed as $\text{ Poisson }(\alpha \pi _I)$. So, with probability $1-\text{ Poisson }(0;\alpha \pi _I) =1 - \exp (-\pi _I\alpha )$ the true sum $\sum _i^\infty z_{ni} \ne \sum _i^I z_{ni}$ and thus we may incorrectly reject or accept a proposal. Conditioned on a desired number of features J, we can further break down the probability of incorrectly rejecting a proposal with $\sum _{i=1}^I(z_{i}^*)<J$ of incorrectly accepting a proposal with $\sum _{i=1}^I(z_{i}^*)=J$ by considering the following possible scenarios:

• 1. $\sum _{i=1}^I z_{i}^* > J$: We reject the proposal. This is always correct.

• 2. $\sum _{i=1}^I z_{i}^* = J$: We accept the proposal. However, if the truncated tail has $\sum _{i=I+1}^\infty z_{i}^*>0$, we should really have rejected. Our decision is correct with probability $P(\sum _{i=I+1}^\infty z_i^* = 0) = \exp (-\pi _I \alpha )$.

• 3. $\sum _{i=1}^I z_{i}^* < J$: We reject the proposal. However, if $\sum _{i=1}^{I}z^*_{i}=J-k$ but the truncated tail has $\sum _{i=I+1}^\infty z_{i}^*=k$, we will really should have accepted. Our decision is correct with probability $1-P(\sum _{i=I+1}^\infty z_i^* = J-\sum _{i=1}^I z_{i}^* )= 1-\text{ Poisson }( J - \sum _{i=1}^I z^*_{i} ; \pi _I \alpha )$.